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Chapter  l 

THE  RESEARCH  PROBLEM 

One  of  the  key  dimensions  of  the  strength  of  the  United 
States  Navy  Is  the  readiness  of  Its  Naval  Air  Forces.  Readiness 
Is  dependent.  In  large  part,  on  the  effectiveness  of  shore-cased 
Navai  Aircraft  Intermediate  Maintenance  Departments  (AIMDs).  The 
AIMDs  are  activities  that  provide  repair  support  to  shore-based 
aircraft  squadrons  in  the  continental  United  States  and  provide 
in-dapth  backup  support  for  the  Carrier  Air  Groups  (CAGs)  based  on 
deployed  aircraft  carriers  around  the  world. 

The  purpose  of  the  present  study  was  to  test  the  Methodology 
for  Generating  Efficiency  and  Effectiveness  Measures  (MGEEM) 
Introduced  and  tested  by  the  Air  Force  Human  Resources  Laooratory 
(AFHRL)  of  the  United  States  Air  Force  In  the  Communication  and 
Navigation  Equipment  Repair  Branch  of  a  shore-based  AIMD  under  the 
Commander,  Naval  Air  Force.  United  States  Pacific  Fleet.  The 
Intent  of  the  research  was  to  determine  whether  the  use  of  the 
MGEEM  Increased  the  effectiveness  of  the  Communication  and 
Navigation  Equipment  Repair  Branch  of  a  shore-based  Naval  AIMD. 
and  to  determine  whether  there  Is  inter-service  transportability 
of  the  MGEEM  by  comparing  the  resuits  of  this  study  performed  in 
the  Navy  with  an  independently-conducted  productivity  study  in  an 
Air  Force  shop  with  an  Identical  organizational  mission. 
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To  achieve  these  two  purposes,  the  following  research 
questions  were  asked: 

1.  Vhat  changes  In  effectiveness  of  the  Coomunicat ion  and 
Navigation  Equipment  Repair  Branch  of  a  shore-cased  Naval  Aircraft 
Intermediate  Maintenance  Department  CAIMD)  were  observed  during 
and  after  implementing  the  MGEEM? 

2.  How  closely  did  the  Key  Result  Areas  c KRAs > .  Indicators 
and  the  subsequent  changes  In  effectiveness  obtained  from  this 
Navy  study  compare  with  those  of  an  Independently-conducted 
implementation  in  an  Air  Force  shop  with  an  Identical 
organizational  mission? 


Pc  1 lnsatlsn 

The  cost  of  modern  weapons  systems  and  the  limitations  of 
budget  and  material  resources  make  it  Imperative  that  industrial 
activities,  such  as  the  shore-based  Naval  AIMDs.  are  run  as 
effectively  as  possible.  The  adnin  i  strati  on  realized  that 
productivity  is  an  important  issue  and.  in  i985.  issued  Executive 
Order  12552.  which  directed  that  ai  l  major  governmental  agencies 
increase  productivity  by  20  percent  by  1992. 

A  standardized  method  of  defining,  quantifying  and  improving 
mission  effectiveness  would  allow  for  a  continuous  ana  uniform 
method  of  measuring  the  relative  standing  of  eacn  AIMD  and  offer  a 
procedure  tor  improving  its  overall  productivity.  This  research 
was  primarily  concerned  with  finding  a  standardized  method  to 
increase  mission  effectiveness,  and  hence  the  productivity,  in  the 
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Communication  and  Navigation  Equipment  Repair  Branch  of  a 
shore-oased  AIMD.  Additionally,  the  research  was  designed  to 
determine  the  Inter-service  transportability  of  the  MGEEM  by 
comparing  It  with  an  Independently-conducted  Implementation  In  an 
Air  Force  shop  with  an  Identical  organizational  mission. 

Because  of  a  policy  of  decentralization  of  authority  in  the 
Navy,  each  AIMD  Is  managed  by  an  AIMD  Officer  with  a  specific 
leadership  style,  and  the  Individual  leadership  style,  to  a  large 
extent,  determines  an  activity's  performance.  The  relative 
standing  of  an  AIMD  with  other  AIMDs  was  not  an  issue  at  the  time 
of  this  study,  since  there  had  been  no  standardized  yardstick 
developed  to  measure  productivity  in  an  AIMD.  In  fact,  finding  a 
standardized  method  of  defining,  quantifying  and  improving  mission 
effectiveness  that,  with  minor  modifications,  would  function  In 
similar  shops  In  all  services  would  greatly  decrease  the  cost  of 
establishing  such  an  organizational  productivity  measurement  and 
enhancement  program  for  the  entire  Department  of  Defense. 

Variables  and  Relationships 

The  research  question  contains  two  variables  that  are  under 
Investigation.  The  Independent  variable.  Methodolgy  for 
Generating  Efficiency  and  Effectiveness  Measures  (MGEEM).  is  a 
tool  developed  and  tested  by  the  United  States  Air  Force  (1986)  to 
establish  a  standard  method  of  measuring  and  enhancing 
organizational  productivity. 
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A  paradign  of  two  observable  elements  (phases)  defines  the 
Independent  variable  as  follows: 

Methodology  for  Generating  Efficiency  8,  Effectiveness  Measures: 

Element  One  (Development  Phase):  Once  familiar  with  the 
target  organization,  the  measurement  coordinator  worked  with  a 
group  of  workers  and  their  Immediate  supervisors  In  a  structured 
group  process  called  the  Nominal  Group  Technique  (NGT)  developed 
Dy  DelCecq,  Van  de  Ven  and  Gustafson  (1975)  to  formulate  and 
prioritize  the  organization's  prlnlcipal  Intended  accompl  isnments. 
called  Key  Result  Areas  (KRAs). 

Using  the  same  NGT  process,  those  supervisors,  together 
with  their  immediate  subordinates,  then  worked  with  the 
measurement  coordinator  to  identify  effectiveness  indicators.  The 
effectiveness  Indicators  are  measures  used  to  determine  the  degree 
to  which  a  specific  KRA  Is  being  accompl  Ished.  They  then  used 
their  judgment,  based  on  knowledge  of  the  job  requirements,  to 
scale  the  indicators  in  terms  of  their  Importance  as  measures  of 
the  KRAs  on  graphs  called  Mission  Effectiveness  Charts  (formerly 
called  Contingency  Charts).  A  computer-generated  report 
consisting  of  graphically-displayed  measures  of  the  Indicators  on 
Mission  Effectiveness  (ME)  Charts  was  then  devised  to  provide 
feedback  to  management  and  workers.  That  feedback  served  to 
inform  all  concerned  just  how  effective  the  target  organization 
was  In  performing  its  principal  objectives,  or  KRAs.  provided 
Information  to  managers  to  help  them  do  their  Jods  better  ana 
provided  performance  information  to  motivate  workers. 
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Element  Two  (Implementation  Phase):  Mission  Effectiveness 
(ME)  Charts  were  then  published  and  distributed  to  the  target 
organization.  Monthly  measurements  of  those  Indicators  were 
gathered  from  the  existing  maintenance  data  collection  system  to 
monitor  any  changes  in  effectiveness  in  the  organization,  and 
feedback  reports  containing  monthly  results  were  distributed  to 
Doth  management  and  workers.  After  the  feedback-only  cycle  was 
complete,  the  additional  productivity  enhancement  of  goal-setting 
with  feedback  was  added  to  observe  its  effect  on  mission 
effect  lveness. 

The  dependent  variable  Is  mission  effectiveness,  which  is 
defined  as  the  extent  to  which  the  outputs  of  the  target 
organization  fulfill  the  needs  of  the  customers. 

The  Research  Approach 

The  research  approach  used  was  descriptive-correlational, 
employing  the  Communication  and  Navigation  Equipment  Repair 
Branches  in  two  major  shore-based  Naval  AIMDs  under  the  Commander. 
Naval  Air  Force.  United  States  Pacific  Fleet  and  the  Communication 
and  Navigation  Equipment  Repair  Shop  at  an  Air  Force  case  located 
in  the  southwestern  United  States.  The  research  sought  to 
determine  whether  the  MGEEM  would  Increase  the  overall  effect¬ 
iveness  In  the  Cammicat ion  and  Navigation  Equipment  Repair 
Branch  of  a  Naval  AIMD.  The  research  concurrently  sought  to 
determine  the  degree  of  relationship  between  this  study's  KRAs. 
Indicators  and  subsequent  changes  In  effectiveness  and  those 
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obtained  In  an  independently-conducted  Implementation  in  a 
Cannunicat ion  and  Navigation  Equipment  Repair  Shop  at  Bergstrom 
Air  Force  Base. 

Using  the  NGT.  the  KRAs  were  determined,  i  Isted  and 
prioritized,  and  indicators  and  ME  Charts  for  those  KRAs  were 
originated.  A  computer-generated  feedback  report  consisting  of 
the  ME  Charts  was  then  developed. 

During  the  Development  Phase  of  the  MGEEM  process,  four 
months  of  past  data  were  collected  to  establish  a  baseline,  and 
then  four  months  of  measurements  were  made  to  observe  any  changes 
in  mission  effectiveness  during  development  of  the  MGEEM.  After 
that,  feedback  alone  and  then  feedback  with  goal -setting  were 
Introduced.  The  KRAs.  indicators  and  ME  Charts  that  had  been 
developed  were  published  monthly  and  distributed  as  feedback  for 
workers  and  management  of  the  branch.  The  essential  research 
question  was  to  determine  whether  this  feedback  alone  and  then 
feedback  with  goal -setting  would  motivate  workers  to  increase 
their  effectiveness  and  would  facilitate  as  we i  i  as  Improve  the 
decision  making  of  managers. 

Criteria  for  the  Data_Source 

To  qualify  as  a  data  source,  the  shore-based  Naval  AIMD  had 
to  be  under  the  Commander.  Naval  Air  Force.  United  States  Pacific 
Fleet  and  had  to  be  classified  as  a  major  AIMD.  A  major  AIMD.  for 
purposes  of  this  study,  is  one  that: 

1.  Serves  at  least  eight  squadrons. 
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2.  Employs  at  least  275  technicians. 

3.  Processes  an  average  of  at  least  300.000  maintenance 
actions  per  year  consisting  of  processing  at  least 
25.000  repairaJble  items. 

Pet  inmoaa 

The  following  definitions  are  listed  to  clarify  terms  used 
throughout  this  paper.  These  terms  pretain  to  the  Naval 
organization  under  study  and  the  MGEEM  process. 

Aircraft  Intermediate  Maintenance  Department  (AIMD)  :  a  Navy 
organization  that  performs  intermediate- 1  eve i  maintenance  on 
aircraft-related  avionics,  airframes,  power  plants  systems  and 
components,  ground  support  equipment  and  other  items  needed  to 
support  Naval  Aviation. 

AIMD  officer:  a  department  head  on  a  naval  air  station  or 
ship  having  the  responsibility  of  managing  the  intermediate-level 
repair  of  aircraft  components. 

Production  control  officer:  a  division  officer  who  airects 
the  efforts  of  the  production  divisions  in  an  AIMD. 

Division  officer:  the  person  in  charge  of  the  functioning  ot 
a  di vision  of  an  AIMD . 

Work  center  supervisor:  the  senior  enlisted  person  on  a 
shift  within  a  given  work  center. 
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Kcv  Result  Areas  (IRAsJ:  the  principal  measurable  aspects  of 
an  organization's  mission;  the  prime  reasons  that  the  organization 
exists. 

Indicators:  the  measurements  used  to  determine  the  degree  to 
which  KFAs  are  being  accomplished. 

flethodolgy  for  Generating  Efficiency  and  Effectiveness 

Measures  (MGEQO:  a  series  of  technologies  by  which 
complete  productivity  measurement  and  enhancement  systems  can  be 
developed  for  any  organization. 

IteaffllCflBgOLJtoa  1  QPfflCIl  t„lc,aa.  1MCI2. ;  a  team  of  members  of  the 
organization  to  be  measured  who  are  Know! edge able  about  the 
organization,  have  good  communication  skills  and  are  opinion 
leaders  by  virtue  of  formal  or  informal  power. 

Nominal  Group  Technique  (NGT) :  a  proven  method  used  by 
groups  to  generate  and  reach  consensus  about  a  complete  set  of 
ideas  or  specifications  on  a  given  subject  or  question.  The 
method  was  developed  by  Deibecq,  Van  de  Ven  and  Gustafson. 

Force-f i e i d  analysis :  a  method  of  displaying  the  forces  in 
favor  of  and  opposed  to  a  specific  topic. 

Mission  effectiveness  measures;  measures  of  the  capacity  for 
doing  the  specific  job  in  question. 

Target  organization  (branch):  the  organization  under  study. 
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Mlaalon  Effectiveness  Charts  (formerly  Contingency  Charts): 


a  method  of  graphically  displaying  organizational  performance, 
which  Is  designed  to  Increase  motivation  when  given  as  feedback  to 
workers  and  to  Improve  decision  making  when  given  as  feedback  to 
managers. 

Weighted  score:  any  score  weighted  proportionately  to 
indicate  the  degree  to  which  It  Is  perceived  to  oe  important  as  a 
measurement  tool . 

Imp  1  cmentat  1  on  Plan :  a  step-by-step  written  procedure  to 
implement  a  given  change  In  an  organization. 

Measurement  Plant  an  outline  of  steps  required  to  be 
accomplished  to  Implement  a  study  of  an  organization. 

Background 

Aircraft  readiness  factors  are  vitally  important  and  are 
reviewed  dally  at  various  levels  of  command  to  monitor  the  current 
status  of  the  Naval  Air  Forces  and  to  determine  the  nature  and  the 
extent  of  deficiencies.  Productivity,  as  a  measure  of  the 
effective  use  of  resources.  Is  an  important  factor  in  readiness, 
since  low  product l v l ty  wi l l  degrade  aircraft  readiness. 

Aircraft  maintenance  has  become  more  crucial  to  aircraft 
readiness  as  aircraft  systems  have  become  more  complex.  As  tnese 
systems  Decome  more  sophisticated,  the  personnel  resources  used  to 
repair  them  have  to  be  used  more  effectively.  Proper  utilization 
or  the  availaDle  personnel  is  a  challenge  for  the  AIMD  Officer. 
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since  the  performance  of  the  AIMD  has  a  lanedlate  and  direct 
hearing  on  aircraft  readiness  of  the  squadrons  served  Dy  the  AIMD. 

At  the  time  of  this  study,  no  objective  method  existed  for 
evaluating  the  effect  of  local  management  on  AIMD  productivity. 

The  administration  had  detected  a  decrease  In  federal  productivity 
and.  in  1965.  President  Reagan  stated  his  view  on  the  Importance 
of  productivity  improvement  to  the  nation  when  he  asked  Congress 
to  pass  a  joint  resolution  declaring  productivity  improvement  as  a 
national  goal . 

The  United  States  Air  Force  has  long  recognized  the 
Importance  of  productivity  improvement,  as  evidenced  by  Air  Force 
Regulation  25-3.  the  Air  Force  Productivity  Enhancement  Program 
(PEP).  This  regulation  establishes  productivity  Improvement  as  a 
orlorlty  in  the  Air  Force,  and  It  acknowledges  the  fact  that  the 
responsibility  for  productivity  Improvement  In  Air  Force 
organizations  rests  squarely  on  the  shoulders  of  the  managers. 

Using  resources  In  the  most  effective  way  possible  to 
accomplish  an  assigned  mission  is  a  manager's  primary  respon¬ 
sibility.  Meeting  this  responsibility  with  limited  oudoets  ana 
expanding  mission  requirements  places  a  premium  on  management 
excellence.  Productivity  growth  is  essential  If  these  management 
challenges  are  to  be  met.  The  MGEEW  evolved  In  the  Air  Force  as 
the  recommended  solution  to  the  productivity  measurement  and 
enhancement  problem.  That  methodology  was  tested  In  many  Air 
Force  organizations,  Including  a  Comunicatlon  and  Navigation 
Equipment  Repair  Shop  in  a  maintenance  squadron  at  Bergstrom  Air 
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Force  Base,  and  significant  increases  in  mission  effectiveness 
were  documented.  In  January  1987,  the  MGEEM  was  Incorporated  Into 
Air  Force  Regulation  25-5  as  the  measurement  component  of  the 
Performance  Measures  Document  (PMD).  This  regulation  empowers  the 
Air  Force  Management  Engineering  Agency  (AFMEA)  to  evaluate  the 
Management  Information  System  <MIS)  In  place  In  every  Air  Force 
functional  area.  For  those  found  to  have  an  Inadequate  MIS.  the 
MGEEM  is  reccmnended  for  use.  On  every  United  States  Air  Force 
Dase  in  the  world,  it  was  expected  that  there  would  be  a  caare  of 
Management  Engineering  technicians  trained  to  Implement  ana 
modify,  as  needed,  the  MGEEM  systems  on  the  local  case. 

Since  both  the  Navy  and  Air  Force  were  faced  with  the  same 
mandate  to  increase  productivity  by  20  percent  by  the  year  1992. 
and  since  the  Air  Force  had  already  field-tested  the  MGEEM  and 
found  It  effective  in  measuring  and  increasing  organizational 
productivity,  it  followed  that  the  Navy  should  test  the  viability 
and  transferability  of  this  proven  methodology  in  Its  own 
organizations.  The  information  gathered  by  this  research  should 
provide  strong  evidence  in  answering  the  question  of  the  inter- 
service  transportability  of  the  MGEEM  and  may  significantly 
contribute  to  designing  a  mu  1 1 1  -service  productivity  measurement 
and  ennancement  program  for  the  Department  of  Defense.  The 
findings  of  this  study  should  also  be  useful  to  A1MD  Officers  in 
increasing  their  ability  to  manage  effectively  ana.  thus,  increase 
the  productivity  In  their  facilities. 
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Summary 

This  chapter  presented  the  background  for  the  research 
problem,  the  research  problem  itself,  and  the  research  questions. 
The  research  problem  was  then  delineated  and  the  neeo  for 
standardized  measurement  of  organizational  productivity  in 
Aircraft  Intermediate  Maintenance  Departments  va9  stated.  The 
variaDles  were  then  named  and  defined. 

The  research  approach  wa9  outlined  and  use  of  the  Nominal 
Group  Technique  to  determine  Key  Result  Areas  (KRAs).  indicators 
and  Mission  Effectiveness  (ME)  Charts  was  discussed  in  that 
outline.  The  criteria  for  the  data  source  were  then  given,  and 
definitions  of  terms  used  in  the  study  were  stated. 

The  background  of  the  study  supported  the  purpose  of  the 
study,  which  was  to  determine  whether  the  use  of  the  Methodology 
for  Generating  Efficiency  and  Effectiveness  Measures  (MGEEM) 
results  in  an  increase  in  mission  effectiveness  in  the 
Communication  and  Navigation  Equipment  Repair  Branch  of  a 
shore-oased  Naval  AIMD.  A  concurrent  purpose  of  the  stuay  was  to 
determine  the  degree  of  inter-service  transoortabi  I  i  ty  of  the 
MGEEM  by  determining  whether  there  was  a  comparable  relationship 
oetween  KRAs,  indicators  and  subsequent  changes  in  effectiveness 
in  an  independently-conducted  implementation  in  a  Communication 
ana  Navigation  Equipment  Repair  Shop  at  Bergstrom  Air  Force  Base. 

The  research  approach  used  in  the  study  was  considered 
descriptive-correlational,  using  the  Communication  and  Navigation 
Equipment  Repair  Branches  of  two  shore-based  Naval  AIMDs  and  the 
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results  of  a  previously  conducted  study  In  an  Air  Force 
Maintenance  Squadron.  The  KRAs.  Indicators  and  resultant  changes 
In  mission  effectiveness  documented  In  this  study  were  compared 
with  the  corresponding  relationships  obtained  frocn  the  Air  Force 
study.  This  comparison  will  serve  as  a  basis  for  a  decision 
making  process  Involving  the  establishment  of  a  multi-service 
productivity  measurement  and  enhancement  program. 


13 


Chapter  2 

REVIEW  OF  SELECTED  LITERATURE 

This  chapter  presents  an  overview  of  the 
literature  related  to  the  current  study.  The  material 
discussed  Is  associated  with  one  or  more  of  the 
following  areas  of  emphasis  explored  in  this  research 
effort : 

cl)  The  background  literature  leading  to  the 
development  of  the  Methodology  for  Generating 
Efficiency  and  Effectiveness  Measures  CMGEEM)  Intro¬ 
duced  by  Tuttle  C1981)  and 

<  2 )  The  background  literature  leading  to  the 
development  of  the  Contingency  Cor  Mission  Effective¬ 
ness)  Measurement  System  developed  by  Pritchard.  Jones 
and  Roth  (190?)  in  conjunction  with  a  study  funded  Dy 
the  Air  Force  Human  Resources  Laooratory  CAFHRL). 

The  research  has  been  organized  as  outlined  In 
these  two  general  areas  In  order  to  provide  a  com¬ 
prehensive  presentation  of  published  findings  that  have 
addressed  problems  in  the  areas  explored  In  the  current 
study . 
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Background  for  the  Methodology  for 

Genttcailiifl  Efficiency  ana 

Siie.c.tl vcnega  fleasucaa 

The  measurement  of  work  effort  has  long  oeen 
considered  an  effective  way  to  Increase  productivity 
(Klug,  1962).  In  a  10-year  study  of  an  Air  Force  work 
measurement  program,  as  applied  to  aircraft  maintenance 
and  supply  functions.  Klug  found  that  labor  efficiency 
rose  from  55  to  89  percent  In  maintenance  and  61  to  08 
percent  in  supply.  Newburn  C1972)  pointed  out  that 
productivity  feedback  reports  to  workers  stimulate 
innovation  and  competition  within  an  organization,  ana 
even  an  imperfect  measure  is  worth  the  effort  as  long 
as  It  stimulates  these  responses. 

Mollenhoff  (1977)  summarized  that  the  development 
ana  Implementation  of  a  work  measurement  system  oasea 
on  sets  of  performance  Indicators  for  Key  Result  Areas 
(KRAs)  and  refined  productivity  standards  was  suitaDle 
for  use  with  workers  performing  complex  icbs.  He 
further  subdivided  KRAs  Into  measurable  units  and  used 
numerical  goals,  ratios,  rating  scale  target  scores  ana 
performance  ranges  to  reflect  reasonable  ana  achievaole 
quantitative  standards  for  productivity  measurement. 

Stewart  (1978)  developed  a  multiplicative  multi- 
attrioute  utility  measure  of  productivity  using  a 
structured  group  process.  The  resulting  measures  were 
used  in  an  aggregated  form  to  represent  overall 
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organizational  productivity  or  In  a  disaggregated  form 
to  monitor  Individual  factors.  As  long  as  the 
organization  accepted  the  measure,  it  appeared  to  De  an 
excellent  tool  for  complex,  di f f leu l t-to-measure 
si tuat Ions. 

After  refining  numerous  attempts  at  developing  Key 
Result  Areas  (KRAs)  ana  Indicators  for  those  KRAs.  the 
Nominal  Group  Technique  (NGT)  developed  by  Deioecq,  Van 
de  Ven  and  Gustafson  (1975)  emerged  as  an  efficient 
method  for  completing  that  process  when  working  with 
groups  that  could  communicate  and  discuss  Items 
together  In  one  setting.  The  NGT  Is  a  group  consensus¬ 
seeking  strategy  consisting  of  the  following  steps 
(Delbecq,  Van  de  Ven  and  Gustafson.  1975:11-13): 

1.  Silent  generation  of  ideas  in  writing: 

2.  Listing  of  ideas,  one  at  a  time.  In 
round-robin  fashion,  that  were  generated 
by  Individual  group  members: 

3.  Discussion  and  clarification  of  the  ‘’raw" 
list  of  Ideas  presented: 

4.  Individual  voting  to  prioritize  items 
from  the  list: 

5.  Further  voting  and  classification  of  items 
and  voting  patterns: 

6.  Revotlng  to  generate  the  final  list. 

The  first  documented  research  application  of  tne 
NGT  to  productivity  measurement  was  in  a  researen 
program  by  Sink  <1977).  conducted  under  the  auspices  of 
the  Ohio  State  University  Productivity  Research  Group. 
Another  researcher  subsequently  discussed  and 
demonstrated  the  app !  I  caDi  l  i  t  y  of  the  NGT  to  the 
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development  of  efficiency  and  effectiveness  indicators 
(Tuttle.  1981). 

In  order  to  access  organizational  performance 

accurately,  a  set  of  measurement  criteria  including 

efficiency  and  effectiveness  is  required.  In  order  to 

have  maximum  utility  for  research  purposes,  criteria 

should  also  meet  the  following  standards: 

i.  Criteria  should,  as  a  whole,  possess  the 

following  characteristics  (Kearney.  1978:11-13): 

Comp  1 eteness.  All  significant  facets  of  the 
organization's  mission  should  be  covered: 
Comparabl  1  1  tv .  Measures  should  remain 
applicable  from  one  time  period  to  another: 
Input  Coverage.  The  output  indicators  should 
cover  all  significant  results  obtained  from 
all  controllable  inputs  to  the  production 
process  (e.g..  the  results  produced  by  the 
work  hours  of  all  people  working  in  the 
organ izat  1  on) : 

Comparabl  1  1  tv.  wl  th  exla-tlnq-qata^_ sourc.es. 

Measures  should  make  maximum  use  of  existing 
data  sources: 

Cost -effect  1 veness .  The  costs  of  measurement 
should  not  exceed  the  benefits  ootained: 
Acceptabl 1 i tv .  The  indicators  should  De 
meaningful  and  acceptable  to  those  wnose 
performance  is  being  measured. 


2.  Criteria  should,  as  a  whole,  have  the 
following  properties  (Tuttle.  Wilkinson  and  Matthews. 
1982:  2-3): 


Consistency .  Be  consistent  across 
organizations  that  perform  slmiliar 
funct ions: 

Comprehens 1 veness.  Cover  all  key  tacets  of 
mission  performance: 
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Heat  -Efficiency  and  Effect  l.venegg  Components. 

Include  multiple  Indicators  of  doth 
efficiency  and  effectiveness. 

3.  Individual  Indicators  should  possess  the 

following  characteristics  (Hurst.  i980:  43-49): 

Val Ldi tv  .  Indicators  should  accurately 
reflect  changes  In  the  organization's 
performance : 

Uniqueness.  Indicators  should  de  relatively 
Independent  of  each  other: 

Understanaabl 1 i tv .  Individuals  Delng 
measured  should  understand  how  their 
performance  Is  reflected  in  the  Indicator 
being  used: 

Con.tro I  i ad  1 1  i  t v  .  Organizational  members 
should  de  able,  through  their  actions  on  the 
jod.  to  produce  changes  In  the  Indicators 
that  are  related  to  their  performance: 

Rel labi 1  1  tv .  Indicators  should  yield 
information  that  is  repeatable  over  time, 
assuming  that  performance  levels  remain  the 
same . 

Many  research  studies  In  the  fie  id  of  productivity 
have  been  conducted  in  the  Air  Force  and  Navy.  A 
research  study  conducted  by  the  Air  Force  (Pritchard. 
Montagno  and  Moore.  i978).  which  explored  the  effect  of 
various  forms  of  job  perfomance  feedback  on 
proauct l vi ty .  suggested  that  feedback  regarding  jod 
performance  has  considerable  potential  for  Improving 
proauct i vl ty .  Fears  are  generated  by  productivity 
measurement  (Tuttle  ana  Sink.  1984).  DUt  strategies  ana 
techniaues  have  been  developed  to  minimize  fear.  Those 
strategies,  if  followed  and  managed  proDerly.  can 
assist  in  effect iveiy  handling  the  change  associacea 
witn  the  Introduction  of  a  productivity  measurement 
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process:  managed  poorly,  they  can  become  a  disaster 
( Brown.  1984). 

A  useful  technique  for  helping  managers  unoerstana 
tne  cnange  process  Is  called  force-field  analysis 
(Mahler.  1974).  Force-field  analysis  is  a  process  of 
analyzing  the  forces  for  and  against  a  change  m 
behavior  by  an  individual  or  group.  A  strategy  for 
increasing  the  forces  for  a  change  in  behavior  is  to 
develop  and  communicate  a  collective  vision  of  the 
organization's  purpose,  philosophy  and  values.  Sin* 
(1984)  argued  that  one  approach  Is  a  strategic 
productivity  planning  process  that  involves 
organization  members  at  all  levels  In  defining  future 
organizational  goals. 

Two  companion  strategies  to  increase  the  forces 
for  a  change  in  behavior  are  (Tuttle  ana  Sink. 

1984:28) : 

(1)  Simply  share  previously  unaisclosea  business 
information.  The  purpose  of  this 
information-sharing  is  to  create  a  sense  of 
trust,  to  educate  subordinates  to  economic 
realities  and  to  subtly  suggest  that  survival 
of  the  organization  and  continued  job 
security  depend  on  maintaining 

compe t i t i veness. 

(2)  Top  management  leadership  must  communicate 
the  message  that  productivity  is  important, 
and  It  must  explain  why. 

Once  the  mood  has  been  set.  the  measurement 
implementation  methodology  is  ready  to  begin.  For  over 
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five  years,  the  Maryland  Center  for  Productivity  ana 
Quality  of  Working  Life  carried  out  research  to  develop 
and  test  a  Methodology  for  Generating  Efficiency  and 
Effectiveness  Measures  <MGEEM>  (Cordry  and  Tuttle. 

1984:  Tuttle,  1981:  Tuttle,  Wilkinson  and  Matthews. 
1982:).  This  methodology  is  highly  participative  and 
makes  extensive  use  of  the  NGT 

(Delbecq.  Van  de  Ven  and  Gustafson.  1975:  Sink.  1983). 

Measurement  is  a  process  that  involves  assigning 
numoers  to  oojects.  events  or  attributes  according  to 
specified  rules  or  procedures.  More  precisely,  a 
measurement  operation  is  a  standardized  rule  that  maps 
each  of  a  set  of  oojects  into  one.  and  only  one.  or  a 
set  of  categories  of  numbers  (Hays.  1967).  According 
to  Nunnally  (1967:3).  "one  of  the  primary  aspects  of 
standardization  requires  that  different  people  who 
employ  the  measuring  instrument,  or  supposedly 
alternative  measures  to  the  same  attribute,  should 
obtain  similar  results."  Hays  <1967)  concurred  by 
arguing  that  the  concept  of  standardization  is  critical 
to  the  measurement  process  and  implies  that  different 
people  who  apply  the  same  rule  to  particular  events  or 
situations  obtain  very  siml  1 iar  results. 

In  the  definition  of  measurement,  it  is  important 
to  note  that  numbers  are  assigned  to  attributes  or 
objects  or  events.  Strictly  speaking,  one  does  not 
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measure  objects,  one  measures  their  attributes  (Emory. 
1985:  Ker linger.  1973:  Nunnaliy,  1967).  Emory  (1985) 
arguea  that  the  Implication  of  this  point  for 
organizational  measurement  Is  that  one  or  a  few 
measurements  of  an  organization  (e.g..  size, 
productivity,  etc.)  should  not  De  used  to  characterize 
or  evaluate  the  whole  organization.  He  Insisted  that 
many  attributes  must  be  measured  if  one  is  required  to 
describe  In  quantitative  terms  an  entity  as  complex  as 
an  organization. 

According  to  Nunnaliy  (1967).  the  use  of  a 
standardized  organizational  measurement  procedure 
offers  many  benefits  to  both  researchers  and  managers. 
One  benefit  is  objectivity.  Use  of  standardized 
measurement  procedures  takes  much  of  the  guesswork  out 
of  ooservaclons  and  allows  Independent  verification  of 
organizational  attributes.  Secondly,  numerical  indices 
provided  by  measurement  procedures  allow  reporting  of 
results  in  finer  detail  and  more  Dreclsely  tnan  wouid 
be  possible  with  subjective  descriptions.  These 
numerical  indices  also  permit  use  of  mathematical  ano 
statistical  analysis  procedures.  Witnout  sucn 
analytical  tools,  organizational  research  would  oe 
seriously  impaired.  Nunnaliy  went  on  to  say  that 
measurement  results  facilitate  communication  between 
researchers,  managers,  members  of  the  organization  and 
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others  who  influence  or  are  affected  by  the 
organization" s  activities.  He  argued  that  unless 
standardized  measurement  procedures  exist,  it  is 
virtually  impossible  to  evaluate  an  organization's 
performance  and  to  communicate  to  an  interested 
audience  that  progress  has  been  made. 

The  type  of  measurement  operations  selected  or 
developed  depends  on  the  attribute  to  be  measured  and 
the  purpose  of  the  measurement.  Four  levels  of 
measurement  are  typically  Identified  by  statisticians: 
nominal,  ordinal,  interval  and  ratio  (Emory.  1985: 

Hays.  1976).  Nunnally  (1967:13-16)  defined  the  four  as 
to  I  lows: 


NOMINAL.  In  nominal  level  measurement, 
numbers  are  used  either  as  labels  or  as  a 
means  of  assigning  people,  objects  or  events 
to  categories.  The  use  of  numoers  in  this 
way  does  not  imply  any  quantitative  meaning 
in  terms  of  amount  of  an  attrioute. 

ORDINAL.  In  ordinal  measurement.  (1)  objects 
of  measurement  are  rank-ordered  from  most  to 
least  with  respect  to  some  attribute.  (2) 
there  is  no  indication  of  how  much  of  the 
attribute  the  measurement  object  possesses 
and  (3)  there  is  no  indication  of  how  far 
apart  the  objects  are  on  the  attribute  being 
measured. 

INTERVAL.  Interval  measurement  procedures 
lead  to  a  precise  ordering  of  objects  of 
measurement  with  respect  to  an  attribute  wnen 
the  distances  between  objects  are  known: 
however,  interval  procedures  do  not  proviae 
any  information  aoout  the  absolute  magnitudes 
of  the  attribute  for  any  object  of 
measurement . 
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RATIO.  Ratio  measurement  includes  all  the 
characteristics  of  interval  measurement,  but 
the  quantities  are  expressed  in  relation  to 
an  absolute  zero,  so  that  absolute  magnitudes 
can  be  inferred. 

When  developing  and  interpreting  measures  of 
productivity,  the  level  of  measurement  produced  Dy  a 
particular  measurement  operation  is  significant.  If  a 
particular  measurement  produces  only  ordinal 
measurement,  but  ratio  Interpretations  are  attempted, 
cney  may  easily  lead  to  inaccurate  conclusions. 

In  addition  to  the  level  of  measurement,  there  are 
also  important  attributes  of  productivity  measures. 
Hurst  (1980:  44-45)  described  nine  characteristics, 
which  he  listed  as  desirable  for  measures  of 
organizational  performance. 

1.  CONTROLLABLE .  The  person  or  group  Deina 
measured  should  have  control  over  the  aspects 
of  performance  measured  by  the  indicator. 

2.  CONGRUENT .  The  performance  measure  for  a 
sub-system  should  be  compatible  with  the 
overall  mission  and  objectives  of  the  larger 
organization  system  of  which  It  is  a  part. 

3.  MEASURABLE .  The  characteristics  snoula  De 
quantifiable  through  procedures  chat  are 
feasible.  In  addition,  a  measurable 
characteristic  should  be: 

a.  Jneou 1  vocal ♦  Not  subject  to 

misinterpretation  and  sensitive  to  actual 
changes  In  performance  that  occur: 

d.  Reproducible.  Similar  performance 
changes  will  produce  similar  measures 
repeatedl  y : 
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c.  Accurate.  Not  subject  to  random  or 
systematic  biases  that  will  Introduce 
measurement  error:  and 

d.  Object  1 ve .  Multiple  observers  of  the 
characteristics  should  agree  on  what  is 
good  performance  and  what  Is  bad 
performance . 

4.  UNDERSTANDABLE .  The  relationsnlp  between  the 
level  of  performance  and  the  measure  is 
understood  by  the  Individual. 

5.  CHQOSABLE.  The  people  being  measured  have  a 
say  In  the  measures  by  which  their 
organization  will  be  judged. 

Hurst  (1980)  acknowledged  that  not  all  of  these 
attributes  of  good  measures  can  be  achieved  at  the  same 
time.  Common  trade-offs  discussed  by  Hurst  <1980) 
included  trade-offs  between  accuracy  and  under- 
standablllty  and  trade-offs  between  congruence  and 
measurabi 1 l ty  . 

Adams  (1968):  Ammons  <1965):  ChobDar  and  Wallin 
< 1984) :  Crawford.  White  and  Maanusson  <1983): 
Dockstader.  Nebeker  and  Shumate  <1977):  Jones  <1985): 
Pritchard.  Bigby.  Belting.  Coverdale  and  Morgan  <1981); 
Pritchard  and  Montagno  <1978):  and  Pritchard.  Montagno 
and  Moore  <1978)  argued  that,  in  addition  to  measure¬ 
ment.  two  of  the  most  establ  ished  techniques  for 
enhancing  productivity  are  feedback  and  goal  setting. 
They  insisted  that  their  research  has  shown  that,  when 
task-related  feedback  is  given  to  individuals  or  groups 
who  are  performing  a  task,  performance  increases  as  a 
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result.  Similarly,  when  performance  goals  are  set  that 
are  acceptable,  specific  and  challenging,  performance 
also  Increases  (Latham  and  Yuki,  1975:  Locke.  1967: 
Locke,  Shaw.  Saar i  and  Latham.  1981;  Steers  and  Porter, 
1974;  Taylor,  Fisher  and  II gen.  1984:  Tubbs.  1986). 

During  the  10  years  prior  to  this  study,  the  Air 
Force  funded  a  series  of  research  and  development 
efforts  focusing  on  the  use  of  feedback  and  goal- 
setting  to  enhance  productivity.  The  study  Dy 
Pritchard.  Montagno  and  Moore  (1978)  Investigated  tne 
effect  of  various  forms  of  job  performance  feedback  on 
productivity.  The  results  suggested  that  feedback 
regarding  job  performance  has  considerable  potential 
for  increasing  productivity.  Tubbs  (1986)  clearly 
indicated  that  worker  participation  in  goal-setting 
further  Increased  productivity,  and  Pritchard.  Jones 
ano  Roth  (1987)  demonstrated  that,  when  feedback  and 
worker  participation  in  goal-setting  are  used  together, 
productivity  is  even  further  enhanced. 
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pgrformMgeJlgAaurgmgnt  Research 

Associated  .with  -the  Contingency 
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Mg  agu  r  emen t  _M  si  hc<ac  1  pq  v 

A  study  oy  Pritchard,  Jones  and  Roth.  <1987; 
examined  the  effects  of  group-level  feedback,  goal- 
setting  and  Incentives  on  productivity  In  Air  Force 
jobs.  The  study  represented  a  more  sophisticated 
procedure  than  previous  studies,  which  demonstrated  the 
positive  effects  of  feedback  and  goal -set t i ng. 

Moreover,  it  dealt  with  more  complex  jobs  having  a 
higher  degree  of  independence.  Research  questions  that 
were  investigated  were:  <a>  Can  a  new  system  of 
productivity  measurement  be  effectively  implemented? 

CD)  Can  feedback  and  worker  participation  in  goal- 
setting  De  used  In  complex  operational  Air  Force  jods 
to  enhance  productivity?  It  was  also  of  Interest  to 
determine  how  these  feedback  and  goal -setting  systems 
affect  work  attitudes  and  how  well  users  liked  tne 
system. 

Pritchard.  Jones  and  Roth  <1987)  constructed  a 
productivity  measurement  system  in  two  steps,  which 
enaDled  them  to  aeal  with  complex.  Interdependent  Jods. 
First,  the  researchers  guided  organizational  members  in 
defining  the  major  tasks  each  organization  was 
responsible  for  carrying  out.  The  members  jointly 
determined  how  to  measure  performance  of  each  task 
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using  the  Methodology  for  Generating  Efficiency  and 
Effectiveness  Measures  as  described  by  Tuttle  and 
Weaver  <1986a.  1986b).  This  effort  resulted  In  13 
measures  for  a  Communication  and  Navigation  Equipment 
Repair  Shop  and  33  measures  for  four  sections  of  a 
supply-related  task.  All  of  these  measures  allowed 
managers  and  workers  to  track  the  major  elements  of 
productivity,  but  did  not  provide  for  Integration  of 
the  various  measures  of  productivity  to  produce  a  “big 
picture"  of  how  effectively  either  organization  was 
functioning.  This  would  have  necessitated  combining 
all  of  the  productivity  measurement  aaca.  ana  comoining 
such  large  amounts  of  productivity  data  Is  difficult 
(Meehl.  1965). 

The  issue  of  combining  the  measurement  data  led  to 
the  second  step.  A  method  was  developed  to  comoine  the 
different  types  of  measures  into  one  overall  measure  of 
how  effectively  the  organization  was  functioning.  This 
was  done  oy  constructing  a  graph  called  a  Mission 
Effectiveness  (ME)  Chart.  An  ME  Chart  relates  eacn 
measure  of  productivity,  called  an  indicator,  to  how 
much  tnat  measure  contributes  to  overall  mission 
effectiveness.  Each  measure  of  productivity  was 
represented  by  an  ME  Chart,  and  each  ME  Chart  was 
developed  throuqn  group  discussions  with  supervisors 
and  managers  . 
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Pritchard,  Jones  and  Roth  ( 1987)  discovered  that* 
although  a  curve  on  an  ME  Chart  could  theoretically  be 
a  linear  relationship  between  a  productivity  measure 
and  overall  effectiveness*  curves  on  ME  Charts  for  the 
five  organizations  measured  were  usually  curvilinear. 
They  demonstrated  that  this  was  an  indication  that 
tasks  do  not  contribute  equally  to  overall 
effectiveness  throughout  the  range  of  performance.  To 
account  for  the  fact  that  all  tasks  are  not  equally 
important,  curves  on  ME  Charts  are  able  to  be  adjusted 
for  relative  importance.  An  ME  Chart  with  a  steep 
slope  represents  an  Important  task,  whereas  a  flat 
slope  represents  a  relatively  unimportant  task. 

Pritchard  and  his  colleagues  (1987)  argued  that 
the  ME  Chart  approach  does  not  attempt  to  remove 
interrelationships  among  the  various  productivity 
measures:  use  of  such  charts  Is  intended  to  maximize 
motivation.  Discussions  with  supervisors  and  managers 
during  Pritchard" s  field  test  suggested  that  tne  ME 
Charts  they  constructed  represented  the  way  that  tney 
actually  fe I t  about  the  relationship  between  perrorm- 
ance  and  mission  accomplishment. 

Pritchard.  Jones  and  Roth  (1987:  3-4)  found  that 
iiE  Charts  are  useful  for  several  reasons: 

(I)  They  permit  the  translation  of  different 

measures  of  productivity  into  a  common  scale 
labeled  'Mission  Effect  I veness . “ 
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(2;  They  capture  management  policy  since  they 

Indicate  the  relative  importance  of  various 
areas  of  work  done  by  the  organization. 

(3;  They  add  an  evaluation  dimension  to 

productivity  measurement  since  they  specify 
how  good  each  level  of  performance  Is  on  each 
Indicator . 

(4)  They  allow  comparison  of  productivity  across 
organ l zat ions . 

Pritchard's  effort  used  a  process  of  consensual 
Judgment  to  construct  ME  Charts.  Supervisors  and 
workers  constructed  and  proposed  ME  Charts  for  man¬ 
agement's  review  and  modification  to  reflect  organ¬ 
izational  policy  and  goals.  Any  modification  was 
completed  in  a  discussion  session  between  management 
and  workers,  and  this  session  was  found  to  be  extremely 
Informative  for  management  and  workers  alike.  During 
Pritchard's  study,  an  average  correlation  level  of  .95 
was  oDtained  between  ME  Charts  developed  tor  different 
snlfts.  The  correlation  demonstrated  that  this  type  or 
productivity  measurement  system  developed  by  one  group 
of  workers  is  In  extremely  good  agreement  with  that  of 
another  group  of  workers  doing  a  similar  set  of  jobs  at 
the  same  location. 

Information  from  the  measurement  system  was  put 
togetner  in  monthly  feedback  reports  and  given  to  all 
workers,  supervisors  and  management  personnel  by  the 
researchers.  These  reports  by  Pritchard.  Jones  and 
Roth  c 1987:  4)  contained  information  including: 

l.  What  the  important  functions  of  the  unit  are: 
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2.  What  should  be  focused  on  to  accomplish  these 
funct 1 ons : 

3.  What  level  of  performance  is  expected  in  each 
area: 

4.  How  much  the  unit  has  done  in  the  last  month 
on  each  of  the  important  indicators: 

5.  How  good  that  level  of  performance  is  on  each 
of  the  indicators  compared  to  the  previous 
month : 

6.  What  an  overal i  index  of  productivity 
resembles  when  aggregated  for  each  of  the 
important  functions: 

7.  What  an  overall  index  of  productivity 
resemoles  when  aggregated  for  the  unit:  and 

8.  How  a  specification  of  the  areas  can  oe 
determined  where  productivity  could  De 
Increased,  and  how  the  priority  for  each  can 
be  set  showing  the  overall  effect  on 
product i v i ty . 

Tuttle  and  Sink  0984)  argued  that  practically  any 
effort  to  measure  and  enhance  productivity  can  expect 
to  be  met  with  some  resistance  because  of  the  perceived 
threats  Inherent  In  being  monitored.  They  outlined  two 
initial  fears:  The  first  was  a  fear  of  staff  reduct¬ 
ions  and  the  second  fear  was  a  concern  that  the 
researchers  would  try  to  teli  them  how  to  do  their  jobs 
oetter.  Once  the  personnel  in  the  target  organization 
understood  that  this  undertaking  was  not  a  manpower 
study  designed  for  staff  reductions,  that  the  product¬ 
ivity  measurement  system  was  to  be  completely  custom- 
fitted  to  their  organization  and  that  they  would  oe 
given  information  about  how  well  they  were  doing  rather 
tnan  advice  on  how  to  do  It.  these  fears  were  great  ly 
reduced . 
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According  to  Pritchard.  Jones  ana  Roth  (1987).  tne 
results  of  giving  feedback  on  this  productivity 
measurement  system  showed  an  average  increase  of  50 
percent  across  five  organizations  compared  with  pre- 
feedback  baseline  levels.  After  five  months  of 
receiving  feedback,  worker  participation  in  goal¬ 
setting  was  added  (each  organization  set  its  own 
oroauctlvlty  goals).  The  average  Increase  in 
productivity  across  five  organizations  with  the 
addition  of  goal-setting  was  74.5  percent  above 
case  line. 

In  addition  to  focusing  on  the  productivity 
effects.  Pritchard.  Jones  and  Roth  (1987)  aiso  assessed 
tne  effects  of  feedback  on  job  attitudes.  There  were 
significant  increases  in  Job  satisfaction  ana  morale 
trom  oaseline  to  feedback  conditions.  In  addition, 
attitudes  toward  the  feedback  system  were  very  positive 
for  Doth  workers  and  supervisors.  Seventy-two  oercent 
of  the  respondents  to  a  questionnaire  assessing 
attitudes  toward  the  system  reported  that  the 
Droauctivity  measurement  system  was  very  well  liked 
overa 1 l . 

Pritcnard.  Jones  and  Roth  (1987)  reported  that 
this  effort  proved  extremely  informative  concerning 
ootn  major  research  questions.  The  resuits  showed  tnat 
cne  new  method  of  measuring  organizational  productivity 
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13  effective  and  that  feedback  and  goal-setting  offer  a 
number  of  Important  benefits  in  complex  operational 
organizations.  Pritchard,  Jones  and  Roth  (1987:  7-8) 
reported  that  benefits  Include: 

1.  The  system  yields  a  wealth  of  productivity 
information.  It  tells  how  well  (how  good  or 
how  bad)  an  organization  is  functioning  on 
each  measure  of  productivity,  how  well  one 
part  of  an  organization  Is  functioning  and 
how  effectively  the  entire  organization  is 
funct  ion ing. 

2.  Building  the  system  causes  an  examination  of 
current  practices.  Throughout  system 
construction,  discussion  of  priorities  and 
decisions  about  work  procedures  are  necessary 
to  build  the  contingencies  and  assemble  them 
into  a  working  productivity  measurement 
system.  This  can  stimulate  changes  in 
current  practices. 

3.  The  system  clarifies  policy.  Various  levels 
of  an  organization  have  input  into  each 
contingency  and  understand  the  views  of  other 
parts  of  the  organization  concerning  what  is 
considered  good  and  bad  performance. 

4.  Workers  see  the  " expect ed“  level  of 
performance  and  what  performance  is 
considered  good. 

5.  Workers  receive  positive  feedback. 

Management  and  the  workers  themselves  see 
when  they  increase  productivity  or  maintain 
high  levels  of  productivity.  They  usually 
find  this  very  motivating. 

6.  Workers  and  mangers  see  the  precise  impact  or 
cnanges  (Increased  effort  or  new  ways  of 
doing  oid  tasks)  on  organizational 

effect l veness. 

7.  Management  can  easily  monitor  the  performance 
of  the  organization. 
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The  results  of  this  research  by  Prltchara  and  ms 
associates  showed  that  feedback  and  goal -setting  can 
Increase  productivity,  since  the  effectiveness  In¬ 
creased  Impressively  In  operational  Air  Force  joos  that 
Involved  Doth  logistics  and  electronics  maintenance. 

In  fact,  Pritchard,  Jones  and  Roth  <1987:  8)  stated 
that  the  results  of  the  research  represented  "the 
largest  increase  In  productivity  as  a  result  of 
feedback  and  goal-setting  ever  obtained  by  them  In  any 
work  environment." 

In  summary,  this  research  showed  that  a  product¬ 
ivity  measurement  system  had  been  developed  that 
contained  maximum  motivational  properties  and  that 
feedback  and  goal -setting  used  by  the  system  Increased 
productivity  Impressively. 

Summary 

This  chapter  summarized  the  literature  selected  as 
a  Dasis  for  the  research.  The  available  material  was 
screened  and  selected  for  Its  applicability  to  military 
and  related  industrial  applications,  its  usefulness  to 
the  researcher  and  Its  relevance  to  other  researcn 
explored  in  productivity  studies  Involving  military  and 
related  Industrial  facilities. 

The  section  of  this  chapter  that  reviewed  the 
background  research  on  which  the  Methodology  for 
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Generating  Efficiency  and  Effectiveness  Measures 


(MGEEM)  model  was  based  featured  the 
by  Tuttle  C1981).  Tuttle  and  Weaver 
and  Pritchard.  Jones  and  Roth  <1987) 


findings  published 
<  1986a.  1986b). 
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Chapter  3 


METHOD 

To  obtain  the  data  required  for  this  study,  two 
major  Aircraft  Intermediate  Maintenance  Departments 
(AIMDs)  were  chosen  as  the  target  organizations  to  be 
examined.  The  AIMD  at  Naval  Air  Station  North  Island. 
Coronado.  California  was  chosen  as  the  facility  to 
implement  the  Methodology  for  Generating  Efficiency  and 
Effectiveness  Measures  CMGEEM).  The  AIMD  at  Naval  Air 
Station  Lemoore.  Lemoore,  California  was  selected  as 
the  facility  to  be  used  as  the  control  group. 

The  data  obtained  as  a  result  of  the  16-month  Navy 
study  and  an  examination  of  the  changes  in  effect¬ 
iveness  observed  in  the  Communication  and  Navigation 
Equipment  Repair  Branch  were  used  to  answer  the 
following  research  questions: 

1 .  What  changes  In  effectiveness  of  the  Comm¬ 
unication  and  Navigation  Equipment  Repair  Branch  of  a 
shore-based  Naval  AIMD  were  observed  during  and  after 
implementing  the  MGEEM? 
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2.  How  closely  did  the  Key  Result  Areas  (KRAs), 
Indicators  and  subsequent  changes  In  effectiveness 
obtained  from  this  Navy  study  compare  with  those  of  an 
independently-conducted  implementation  in  an  Air  Force 
shop  with  an  identical  organizational  mission? 

Co  I \ ect 1  on  of  Data 

The  research  was  concerned  with  evaluating  a 
standardized  method  of  Increasing  mission  effectiveness 
and  thus  organizat  ional  productivity  in  shore-based 
Naval  AIMDs.  Two  f u 1 1 y-funct  ional  major  AIMDs  under 
the  Commander,  Naval  Air  Force.  United  States  Pacific 
Fleet  were  used  in  the  study. 

The  research  required  three  data  sources:  One 
experimental  facility  and  one  control  facility  for  the 
Navy  study  and  an  independently-conducted  study  per¬ 
formed  by  the  Air  Force.  The  MGEEM  process  that  was 
investigated  is  a  multi-step  procedure  employing  a 
measurement  facilitator  who  leads  a  target  organization 
through  a  structured  group  process  called  the  Nominal 
Group  Technique  (NGT) ,  which  was  developed  by  Deibecq. 
Van  de  Ven  and  Gustafason  0975).  The  NGT  was  used  to 
define  and  prioritize  the  principal  Intended  objectives 
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of  the  target  organization,  which  are  called  Key  Result 
Areas  (KRAs).  A  Measurement  Development  Team  (MDT) 
consisting  of  top  level  managers  along  with  their  sub¬ 
ordinates  was  used  to  develop  the  KRAs.  Once  the  KRAs 
were  defined,  the  MDT  was  used  to  identify  indicators, 
which  were  used  to  periodically  assess  whether  the  KRAs 
were  being  accomplished,  and  Mission  Effectiveness  (ME) 
Charts  were  then  developed  and  programmed  on  a  computer 
to  periodically  provide  feedback  on  the  organization's 
performance  on  Indicators  to  both  the  management  and 
the  workers.  This  feedback  alone,  and  later  feedback 
coupled  with  goa I -sett  ing,  were  studied  for  their 
impact  on  mission  effectiveness. 

The  entire  MGEEM  process  was  implemented  in  the 
Communication  and  Navigation  Equipment  Repair  Branch 
located  at  Naval  Air  Station  North  Island.  Coronado. 
California  AIMD.  The  KRAs  and  Indicators  developed  at 
the  Naval  Air  Station  North  Island  AIMD  were  published 
and  distributed  as  policy  and  implemented  in  the  Comm¬ 
unication  and  Navigation  Equipment  Repair  Branch.  A 
computer-generated  feedback  program  to  display  ME 
Charts  and  provide  productivity  enhancement  was 
developed,  and  feedback  reports  were  made  to  both 
managers  and  workers  to  observe  changes  in  mission 
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effectiveness  based  on  feedback.  After  the  Feedback 
Phase  was  completed,  goal -setting  was  added  as  an 
additional  enhancement  to  observe  changes  in  mission 
effectiveness  based  on  feedback  coupled  with  goal- 
setting. 

The  Naval  Air  Station  Lemoore,  Lemoore,  California 
AIMD  acted  as  the  control  group  and  was  not  involved 
with  any  phase  of  the  MGEEM  process.  The  KRAs  and 
Indicators  developed  at  Naval  Air  Station  North  Island 
were  used  as  items  to  be  measured  at  both  locations  so 
that  the  gathered  data  were  able  to  be  compared.  Four 
months  of  historical  data  were  gathered  at  both  AIMDs 
Initially  to  establish  a  baseline.  Monthly  measure¬ 
ments  were  then  taken  at  the  AIMDs  for  four  months 
during  the  Development  Phase,  for  four  months  during 
the  Feedback  Phase  and  for  four  more  months  during  the 
Goal -Setting  Phase. 

The  data  from  both  Naval  AIMDs  and  the  Air  Force 
shop  were  then  analyzed  to  determine  (1)  whether  the 
MGEEM  process  was  associated  with  a  change  In  mission 
effectiveness  at  Naval  Air  Station  North  Island  and  <2; 
whether  the  KRAs.  Indicators  and  resultant  changes  In 
effectiveness  obtained  from  this  Navy  study  were 
similar  to  those  obtained  In  an  Independently-conducted 


38 


Implementation  In  an  Air  Force  3hop  with  an  Identical 
organizational  mission.  The  AIMDs  used  In  this  study 
are  described  In  Table  1. 


Table  1 

Description  of  the  Aircraft  Intermediate  Maintenance 
Departments  (AIMDs)  That  Constitute 
The  Research  Population 


MAJOR 

Number  of 

Number  of 

Annual  Repair 

AIMD 

Squadrons 

Technicians 

Act i ons 

Supported 

Qnp 1 oyed 

Performed 

Lemoore 

13 

653 

27.279 

lorth  Island 

21 

705 

35.979 

The  study  design  specified  a  16-month  evaluation 
period  to  minimize  Inaccuracies  caused  by  the  supported 
squadrons'  dl f f er l ng  ml sslons ,  deployment  schedules  and 
operating  environments.  Some  unavoidable  influences, 
such  as  geographic  operational  area  and  specific  mis¬ 
sion  of  the  supported  squadrons,  were  kept  to  a 
minimum,  but  were  not  considered  to  be  relevant 
factors,  due  to  the  fact  that  only  changes  In  mission 
effectiveness  from  a  given  basel ine  for  each  AIMD  were 
being  observed. 
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Lngtrumentat  i  on 


The  measurement  method  used  to  assess  organ¬ 
izational  productivity  Is  the  result  of  applying  the 
MGEEM  process  to  Identify  KRAs.  Indicators  and  develop 
their  resultant  Mission  Effectiveness  (ME)  Charts.  The 
methods  used  as  enhancements  were  feedback  and  goal- 
sett 1 ng. 

This  approach  to  measuring  productivity  was  de¬ 
veloped  as  a  result  of  funding  supplied  by  the  Air 
Force  Human  Resources  Laboratory  CAFHRL)  and  was  field- 
tested  In  1906  In  a  Communication  and  Navigation 
Equipment  Repair  Shop  and  in  four  branches  of  a  Supply 
Squadron  at  an  Air  Force  base  located  in  the  south¬ 
western  United  States.  The  evaluation  of  the 
productivity  measurement  system  was  a  part  of  a  larger 
project,  which  focused  on  interventions  of  feedback, 
goal-setting  and  incentives  as  techniques  for  enhancing 
organizational  productivity. 

To  assess  interventions,  four  months  of  historical 
data  were  gathered  to  establish  a  baseline  before  the 
Interventions  were  Implemented.  If  no  historical  data 
were  present,  the  most  current  month's  data  were  used 
to  establish  the  base  1  lne  from  which  mission  effect- 
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lveneas  changes  due  to  the  Interventions  were  aDle  to 
oe  evaluated. 

The  frequency  for  reporting  mission  effectiveness 
data  for  indicators  within  each  KRA  was  established  as 
one  month  In  order  to  parallel  the  work  cycle.  The 
data  were  obtained  monthly  primarily  from  the  In-place 
maintenance  data  collection  system.  The  computer¬ 
generated  feedback  program  developed  to  report  results 
was  capable  of  taking  raw  Input  data  and  producing  a 
weighted  feedback  chart  called  a  Mission  Effectiveness 
(ME)  Chart.  ME  Charts  display  the  Judgments  of  members 
of  the  work  center  concerning  the  Impact  of  various 
levels  of  performance,  displayed  as  Indicators,  on  the 
work  center's  overall  mission  effectiveness.  Thus, 
there  is  one  ME  Chart  for  each  indicator.  Each  ME 
Chart  shows  a  given  Indicator  on  the  X-axls  and  always 
shows  mission  effectiveness  on  the  Y-axis:  see  Figure 
3. 

Curves  on  ME  Charts  account  for  the  fact  that  some 
Indicators  of  productivity  are  more  important  than 
others:  therefore,  the  computer-generated  feedback 
report  could  aggregate  to  obtain  the  total  effective¬ 
ness  score  by  adding  the  effectiveness  scores  for  each 
ME  Chart.  This  score  was  compared  with  baseline  and 
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the  previous  month's  effectiveness  score  to  determine 
the  extent  of  change  In  mission  effectiveness,  hence 
productivity. 

Procedure 

To  obtain  permission  to  conduct  the  study  and  to 
schedule  on-slte  visits  with  the  AIMDs  selected  as  data 
sources,  the  researcher  was  assigned  In  a  Temporary 
Additional  Duty  (TAD>  status  as  the  measurement  facil¬ 
itator  for  productivity  In  the  Productivity  Branch  of 
the  Aircraft  Maintenance  Department  assigned  to  the 
Commander.  Naval  Air  Force,  United  States  Pacific  Fleet 
Staff.  Telephone  calls  to  the  AIMD  Officers  estab¬ 
lished  the  Initial  consent  to  conduct  the  study:  a 
follow-up  call  established  a  mutually  convenient  date 
for  conducting  the  Development  Phase  of  the  MGEEM 
process.  The  procedure  followed  throughout  this  study 
parallels  the  procedure  suggested  by  Tuttle  and  Weaver 
(1986a.  1986b)  In  their  guides  for  measurement  facil¬ 
itators  In  the  use  of  the  MGEEM  process. 

At  the  pre-arranged  time,  date  and  place,  an  in¬ 
itial  meeting  with  the  target  Naval  Air  Station's  AIMD 
Officer  marked  the  formal  beginning  of  the  MGEEM 
Implementation.  At  that  meeting,  time  was  devoted  to 
establish  a  rapport  with  the  AIMD  Officer  and  to  assure 
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him  that  the  purpose  of  the  MGEEM  Implementation  was  to 
develop  a  tool  that  would  measure  and  Improve  the 
productivity  of  the  AIMD.  The  structured  Interview  In 
Appendix  A  employed  by  Tuttle  and  Weaver  Cl 986b)  was 
used  for  assistance  In  the  Initial  meeting.  In  the 
meeting,  the  AIMD  Officer  also  received  an  overview 
briefing  on  the  MGEEM  Development  and  Implementation 
Phase,  and  the  major  steps  of  the  process  were  ex¬ 
plained  along  with  a  suggested  timetable  for  Implement¬ 
ation.  All  questions  were  answered  and  the  AIMD 
Officer  was  assured  that  the  MGEEM  measurement  facil¬ 
itator  was  there  to  benefit  the  organization,  not  to 
evaluate  or  threaten  It. 

A  review  of  existing  in-house  documentation  was 
then  made  to  ensure  that  the  measurement  facilitator 
was  familiar  with  the  target  organization.  Examples  of 
documentation  Included  In  this  review  are  listed  in 
Appendix  B. 

Another  source  of  Information  about  the  target 
organization  was  obtained  through  a  site  visit  equiv¬ 
alent  to  the  one  suggested  by  Tuttle  and  Weaver 
(1986b).  The  measurement  facilitator's  primary  concern 
in  this  site  visit  was  to  learn  more  about  the  target 
organization.  Examples  of  specific  l  terns  Included  In 
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the  site  visit  are  included  in  Appendix  C.  The 
facilitator  then  focused  on  identifying  the  work 
center's  major  products  and  services,  as  wel 1  as 
developing  a  general  understanding  of  the  work  flow 
i nvo 1 ved. 

It  would  have  been  possible  for  the  facilitator  to 
do  an  adequate  Job  of  conducting  the  MGEEM  implement¬ 
ation  without  having  seen  the  work  site,  but  there  were 
distinct  advantages  to  such  a  visit.  One  advantage  was 
the  opportunity  to  establish  a  rapport  with  members  of 
the  work  center.  If  the  facilitator  was  seen  as  caring 
enough  to  understand  how  the  organization  worked, 
members  tended  to  cooperate  more  fully  and  were  more 
willingly  to  cooperate  with  the  requirements  of  the 
research.  Another  advantage  is  that,  through  seeing 
the  work  site,  the  facilitator  was  able  to  communicate 
more  effectively  in  the  group  sessions  to  follow  and 
was  able  to  better  understand  participants'  comments 
because  he  had  become  familiar  with  the  pattern  of  the 
work  flow. 

The  facilitator  then  used  a  systems  diagram,  or 
Input-output  approach,  suggested  by  Tuttle  and  Weaver 
< 1986b)  to  illustrate  the  multiple  organizations  that 
were  Involved  when  determining  the  mission  effective- 
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neas  of  the  target  organization.  A  system  la  a  get  of 
Interrelated  components  that  have  a  common  purpose  or 
goal.  A  Navy  organizational  system  could  be  an  air 
wing,  a  squadron,  a  maintenance  department  or  even  a 
branch  within  a  maintenance  department.  Systems  can  be 
small  or  large,  but  they  all  have  a  set  of  Interrelated 
components,  such  as  people,  materials,  equipment  and 
organizational  entitles  that  share  a  common  mission. 
Systems  are  defined  In  terms  of  their  Inputs,  outputs, 
goals  and  Interactions  with  their  environment  across 
system  boundaries.  Figure  1  pictures  a  system  defined 
by  Its  boundary,  which  separates  It  from  Its  environ¬ 
ment  . 

As  a  general  rule.  Inputs  to  any  organization  can 
be  grouped  Into  labor,  material,  capital  and  energy. 
Outputs  are  defined  as  the  products  or  services  that 
the  system  produces  to  achieve  Its  purpose.  The  main 
outputs  of  a  maintenance  organization  are  the  number  of 
pieces  of  equipment  repaired,  personnel  trained,  parts 
fabricated  and  reports  submitted. 

The  system's  boundary  separates  the  system  from 
its  environment.  Although  the  concept  of  a  boundary  is 
theoretical,  it  has  practical  implications,  since  a 
manager  is  able  to  control  what  goes  on  in  the  system 
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while  having  little  control  over  what  occurs  outside 
the  system.  Once  a  system  Is  defined  by  specifying  Its 
boundaries.  Its  environment  consists  of  anything  that 
is  not  contained  within  the  system.  More  detailed 
systems  diagrams  than  shown  In  Figure  l  can  be  prepared 
after  going  through  an  organizational  familiarization 
process . 

The  facilitator  used  the  concept  of  a  systems 
diagram  to  better  understand  the  target  organization. 
The  diagram  for  the  system  under  study  is  shown  in 
Figure  2.  It  can  be  seen  that  Inputs  are  derived  from 
the  environment,  and  the  system  uses  these  Inputs  to 
produce  outputs,  which  are  then  disbursed  to  the 
envl ronmen t . 
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Organizational  Diagram  for  the  Communication  and 
Navigation  Equipment  Repair  Branch 
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After  developing  the  systems  diagram,  it  was 
explained  to  the  AIMD  Officer  that  there  are  many 
reasons  why  organizations  decide  to  measure  their 
productivity.  In  general,  these  motives  are  either 
organizational  control  or  organizational  Improvement. 
The  motive  of  organizational  control  Is  that  organ¬ 
izations  want  to  generate  measures  to  satisfy  someone 
else,  such  as  a  higher  command  or  a  controlling  agency. 
The  motive  of  measurement  for  organizational  Improve¬ 
ment  has  a  different  focus  In  that  the  managers 
themselves  create  measures  as  guides  to  assist  In 
Ident  1  fylng  methods  to  Improve  organizational  effect¬ 
iveness.  Thus,  contro 1 -or iented  measurement  Is  used  by 
organizations  to  Justify  themselves  to  other  organ¬ 
izations  in  their  environments,  while  Improvement- 
oriented  measurement  provides  information  to  enable 
organizations  to  better  manage  themselves.  (This 
distinction  Is  made  here,  since  It  was  explained  that 
the  MGEEM  process  should  be  seen  as  an  Improvement- 
oriented  approach  to  management.) 

A  basic  purpose  of  developing  Improvement -or l entea 
measurement  Is  that  it  provides  an  organization  with  a 
method  of  Initiating  productivity  Improvement.  It  was 
explained  by  the  researcher  to  the  AIMD  Officer  and  MDT 
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that  the  relevant  measures  to  be  developed  by  the  MGEEM 
would  not  be  those  that  fall  under  the  control  of 
external  organizations,  but  would  be  those  controlled 
by  the  AIMD  Itself.  Through  use  of  Improvement- 
oriented  measurement.  It  was  shown  that  the  AIMD 
Officer  and  his  subordinates  could  assess  current 
performance  and  take  corrective  action. 

The  facilitator  then  worked  with  the  AIMD  Officer 
and  appropriate  branch  personnel  to  Identify  potent¬ 
ially  helping  and  restraining  forces  associated  with 
the  goal  of  Implementation  of  the  MGEEM.  Tuttle  and 
Weaver  < 1986b)  also  suggested  this  process  In  their 
facilitator's  guide.  The  reason  for  outlining  these 
forces  Is  to  enable  the  facilitator  to  put  together  a 
strategy  to  strengthen  the  forces  for  success  and  to 
minimize  forces  working  against  success.  A  sample  of 
these  forces  Is  shown  In  Table  2. 
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Tab  I e  2 


Analysis  of  the  Forces  For  and  Against 
MGEEM  Implementation 


Forces  For 

imp  1 ementat ion 

of  the  MGEEM  process 


Forces  Against 
imp i ementat  ion 
of  the  MGEEM  process 


AIMD  Officer's  support - > 

Desire  to  improve - > 

Division  Officers  support  —  > 

Supervisor's  support - > 

Desire  to  have  a  voice - > 


< — Fear  of  more  paperwork 
< — Fear  of  more  workload 
< — MGEEM  process  unknown 
< — Fear  of  monitoring 
< — Fear  of  losing  freedom 


Once  the  force  field  analysis  summarized  in  Taole 
2  had  been  completed,  the  results  served  to  guide  the 
development  of  an  implementation  plan.  A  sample  im¬ 
plementation  plan  format  is  presented  in  Appendix  D. 

The  imp iementat ion  plan  was  then  communicated  over 
a  short  period  of  time  to  al  i  members  of  the  Commun¬ 
ication  and  Navigation  Equipment  Repair  Branch  of  the 
AIMD  from  the  top  down  as  suggested  by  Tuttie  and 
Weaver  (1986a).  Tuttie  and  Weaver  C1986b:9)  aiso 
suggested  that  caution  should  be  taken  to  avoid  certain 
common  misconceptions  and  errors: 
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1.  The  perception  by  the  workers  that  the 
measurement  effort  was  being  forced  on  the 
organization  and  does  not  have  the  full 
support  of  branch  and/or  section  chiefs.  To 
avoid  this  misconception,  strong,  continuous 
and  visible  management  support  for  the  MGEEM 
process  was  essential. 

2.  The  perception  that  the  process  was  designed 
to  require  more  work  from  members  of  the 
organization.  To  avoid  this  mlsconcept Ion . 

It  was  emphasized  that  Improvement,  not 
control ,  Is  the  main  purpose  of  the  MGEEM. 
Management  made  their  endorsement  of  this 
purpose  to  all  members  of  the  organization. 
Management  also  stressed  that  the  MGEEM  would 
Identify  areas  where  performance  could  be 
Improved  by  working  smarter,  not  harder. 

3.  Neglecting  to  recognize  the  legitimate 
organizational  concerns  and  limitations 
during  the  Implementation  of  the  MGEEM 
process.  For  Instance,  It  Is  a  mistake  not 
to  pay  enough  attention  to  the  fears  of 
organization's  members.  Implementation  had 
to  include  a  sincere  effort  to  give  people  a 
chance  to  voice  their  fears  and  concerns  and 
to  have  those  concerns  answered  honestly. 
However.  Implementation  was  not  allowed  to 
drag  on  until  each  minor  Issue  was  resolved. 

4.  Failure  to  ensure  that  the  target 
organization  had  been  clearly  defined  by  the 
measurement  facilitator.  To  avoid 
misunderstandings,  the  facilitator  had  a 
clear  conception,  displayed  In  a  systems 
diagram,  of  what  organizational  units  and 
functions  were  within  and  beyond  the  scope  of 
the  measurement  activity.  This  understanding 
was  shared  and  agreed  to  by  the  commander  and 
management  of  the  target  organization. 

The  next  step  Involved  forming  a  Measurement 
Development  Team  CMDT)  similar  to  that  suggested  by 
Tuttle  and  Weaver  <1986a,  1986b).  Since  the  MGEEM 
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Involves  participative  methodologies,  the  facilitator 
worked  with  the  AIMD  Officer  to  ensure  that  the  MDT 
members  were  carefully  selected.  MDT  members  needed 
sufficient  knowledge  of  the  organization  and  Its 
primary  work  process  to  contribute  Ideas  and  make 
meaningful  Judgments.  Furthermore,  they  had  to  be  able 
to  communicate  their  ideas  to  those  with  whom  they  work 
and  were  required  to  be  key  members  or  opinion  leaders 
by  virtue  of  their  formal  or  informal  power. 

Two  MDTs  were  formed  by  Tuttle  and  Weaver  (1986a. 
1986b):  however.  It  was  found  beneficial  to  form  only 
one  MDT  In  this  Navy  study,  since  the  Communication  and 
Navigation  Equipment  Repair  Branch  is  a  relatively 
small  part  of  the  AIMD.  The  MDT  members,  consisting  of 
the  division  officer,  senior  personnel  and  opinion 
leaders  In  the  work  center,  first  defined  the  broad 
measurable  facets  of  the  branch's  mission  (called  the 
KRAs).  Once  the  KRAs  were  developed,  the  specific 
Indicators  and  Mission  Effectiveness  (ME)  Charts  for 
each  Indicator  were  developed. 

From  the  viewpoint  of  the  external  environment. 
KRAs  are  the  reasons  that  the  organization  exists.  The 
MDT  needed  to  be  briefed  on  the  meaning  of  a  KRA.  This 
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Initial  briefing  paralleled  the  one  suggested  by  Tuttle 
and  Weaver  <1986b)  and  Is  outlined  In  Appendix  E. 

The  facilitator  used  the  NGT  to  simplify  the  KRA 
development  process.  The  following  question  was  askedi 
“What  output  or  results  does  the  Navy  expect  this 
branch  to  accomp 1 1 sh?"  Members  of  the  MDT  were  asked 
to  write  answers  to  this  question  on  a  sheet  of  paper. 
Once  written,  all  of  the  answers  tproposed  KRAs;  were 
posted  on  chart  paper  around  the  room.  The  proposed 
KRAs  were  discussed,  modified  and  consolidated  until  a 
preliminary  listing  of  KRAs  had  been  generated.  The 
primary  reason  that  all  of  the  Ideas  were  listed, 
discussed,  modified  and  consolidated  was  to  attempt  to 
consider  everyone's  Inputs:  a  further  reason  was  to 
reduce,  consolidate  and  refine  the  KRAs,  which  set  the 
stage  for  voting  to  reach  consensus  on  the  final  set  of 
KRAs  among  members  of  the  MDT. 

The  1  Istlng  was  then  discussed  and  voted  on  to 
reduce  and  prioritize  the  number  of  KRAs  on  the  list. 

As  a  result  of  the  first  vote,  the  KRAs  began  to  emerge 
that  were  considered  the  most  Important  Dy  members  of 
the  MDT.  The  condensed  list  Is  shown  In  Appendix  F. 
After  more  discussion  and  modification,  a  second  vote 
was  conducted  to  further  reduce  and  prioritize  the  list 
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of  KRAs  in  order  of  their  importance.  The  final  KRAs 
are  shown  In  Table  3.  The  MDT  needed  to  vote  only 
twice  to  reduce  the  KRAs  to  a  managable  number,  but 
another  vote  would  have  been  used  if  necessary. 

Table  3 

Final  KRA  Listing  of  the  Communication 
and  Navigation  Equipment 
Repair  Branch 


KRA  *  1.  Customer  .Satisfaction 

Describes  the  organization's  primary  means  of 
measuring  the  service  it  is  required  to  provide. 

KRA  #  2.  teUMlOfl 

Describes  the  organization's  ability  to  meet  the 
needs  of  the  assigned  workers. 

KRA  #  3.  Training 

Describes  the  organization's  ability  to  meet  its 
training  needs. 

KRA  #  4.  Safety 

Describes  the  organization's  ability  to  conduct 
its  Job  in  a  safe  manner. 
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After  the  KRAs  were  developed,  the  MDT  developed 
what  they  considered  the  best  performance  Indicators 
for  each  KRA,  as  suggested  by  Tuttle  and  Weaver  <  1986a, 
1986b>.  While  deciding  on  the  best  Indicators  for  each 
KRA,  use  of  existing  statistical  data  available  from 
the  maintenance  data  collection  system  was  encouraged. 
This  proved  extremely  helpful,  since  the  MDT  members 
were  familiar  with  the  meaning  of  the  maintenance  data 
that  were  presently  being  collected,  and  new  data 
collection  Durdens  on  the  organization  were  kept  to  a 
ml  n imum. 

The  first  KRA  for  which  Indicators  were  to  be 
developed  was  purposely  chosen  to  be  the  easiest  one. 
This  served  to  familiarize  MDT  members  with  the  process 
of  developing  indicators.  Formulating  indicators  for 
other  KRAs  was  then  patterned  after  the  process  used  on 
the  first  one. 

While  developing  Indicators,  members  of  the  MDT 
were  asked  to  write  down  all  the  potential  Indicators 
that  they  could  think  of  for  the  KRA.  The  Indicators 
were  then  listed  and  discussed,  modified,  consolidated 
and  voted  on  In  the  same  manner  as  the  KRAs  were  in 
their  development  stage,  until  the  number  of  Indicators 
for  each  KRA  was  reduced  to  no  more  than  six  per  KRA. 
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Thl3  wag  done  to  ensure  that  the  measurement  system 
measured  only  the  most  important  Indicators:  measuring 
more  than  six  Indicators  per  KRA  was  considered  to  be  a 
paperwork  burden  by  the  MDT .  The  final  KRAs  and 
indicators  are  listed  in  Table  4. 
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Tab 1 e  4 


Final  KRA  and  Indicator  Listing  for  the 
Communication  and  Navigation 
Equipment  Repair  Branch 


KRA  #  1.  Customer  Satisfaction 

Indicator  A:  Y-Code  rate:  Percentaae  of  repaired 
equipment  that  did  not  function 
immediately  after  Installation. 

Indicator 

a= 

Backlog:  Number  of  items  that  were 
awaiting  maintenance  <AWM>  or 
awaiting  parts  (AWP). 

Indicator 

Exrep  backlog  items:  Number  of  items 
that  were  AWM  or  AWP  for  expeditious 
repair . 

indicator 

Q: 

RFI  rate:  Percentage  of  equipment 
brought  in  for  repair  that  was 
actual ly  repaired. 

Indicator 

E: 

Turn  around  time:  Average  processing 
time  taken  to  repair  items. 

Indicator 

E: 

Broad  Arrow/TED  reports:  Number  of 
test  benches  that  were  Inoperative  or 
operating  in  a  degraded  condition. 

KRA  #  2.  Motivation 

Indicator  A:  Recoanltion:  Number  of  DeoDle  formally 
recognized  or  rewarded  per  month. 

Indicator. 

Morale  Index:  Average  of  feedback 
scores  on  a  quarterly  morale  survey. 

Indi cator 

Production  Effort:  Percentage  of  work 
accomplished  by  assigned  personnel. 

indlsaiflc. 

J}: 

Negative  personnel  Indicators:  Number 
of  negative  reports  received  on 
personnel  in  the  work  center. 

ladicat-on 

E: 

Quarterly  Retention:  Percent  of  chose 
eligible  who  reenllst  or  extend. 
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Table  4  (continued) 


KRA  #  3.  Training 

Indicator  A: 

QJT:  Average  number  of  on-the-job 
training  hours  documented  per  month 
per  person. 

Indicator  B: 

PQS:  Percent  of  assigned  Personnel 
Qualification  Standards  that  have 
been  completed. 

Indicator  C: 

Rate  Training:  Average  hourly  amount 
of  training  completed  In  Navy 
professional  jobs. 

Indicator  D: 

Formal  Billet  Training:  Average  hourly 
amount  of  training  received  in 
specific  Job  assignment. 

KRA  #  4.  Safety 

Indicator  A: 

Accidents:  Number  of  reported 

accidents  at  work. 

Indicator  B: 

Safety  Violations:  Number  of  safety 
violations  reported  at  work. 

Indicator  C: 

Repeat  Discrepancies:  Number  of  repeat 
discrepancies  on  the  Safety  Audit. 

The  next  phase  of  the  measurement  process  captured 
the  MDT's  Judgment  on  the  re  1  active  Importance  of  each 
indicator  and  on  its  Impact  on  the  Communication  and 
Navigation  Equipment  Repair  Branch's  overall  level  of 
performance.  The  resulting  graph  is  referred  to  as  a 
Mission  Effectiveness  (ME)  Chart.  ME  Charts  were 
developed  by  the  Air  Force  Human  Resources  Laboratory 
CAFHRL)  specifically  for  this  purpose. 
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Each  ME  Chart  employs  a  two-dimensional  (X,Y>  axis 
and  creates  a  unique  transformation  for  each  indicator. 
On  the  Y-axis,  overall  mission  effectiveness  Is 
measured  on  a  scale  in  Increments  of  10  from  0  to  +100 
on  the  top  and  0  to  -100  on  the  bottom.  There  is  one 
ME  Chart  for  each  Indicator,  and  the  different  levels 
of  possible  performance  on  the  indicator  are  relected 
on  the  X-axis  of  the  graphs.  For  each  given  indicator, 
the  relationship  between  performance  on  the  Indicator 
and  its  Impact  on  mission  effectiveness  is  plotted. 

To  construct  each  ME  Chart,  group  discussion  was 
used  by  the  MDT  to  determine  points  on  the  graph.  The 
first  point  determined  was  the  expected  level  of  per¬ 
formance  (or  zero  point),  which  was  determined  by 
asking  the  question,  "What  is  the  level  on  this 
indicator  that  has  no  impact  on  mission  effectiveness?" 
Another  way  to  put  the  question  was  to  ask,  '‘What  is 
the  level  on  this  Indicator  that  will  not  cause 
management  to  commit  more  resources  in  order  to  affect 
mission  effectiveness?"  The  coordinate  for  this  is 
zero  on  the  Y-axis  and  it  represents  the  expected  level 
or  zone  of  performance  on  the  X-axis. 
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Once  the  expected  level  or  zone  was  determined,  the 
points  of  possible  maximum  and  minimum  performance  on 
the  indicator  were  determined,  which  “anchored"  the 
values  on  the  graph. 

Intermediate  points  of  effectiveness  were  then 
determined  b y  discussion,  and  the  resulting  graph  was  a 
possibly  non-linear  representation  of  the  effectiveness 
produced  by  different  levels  of  performance  on  each 
indicator.  Consensus  through  discussion  was  then  used 
to  rank  the  maximum  of  each  indicator  In  terms  of  Its 
contribution  to  the  overall  productivity  of  the  unit. 
The  maximum  with  the  greatest  contribution  was  auto¬ 
matical  ly  given  an  effectiveness  value  of  +i00,  and  the 
group  rated  the  other  maximum  vaiues  relative  to  the 
most  Important  one.  For  example,  if  the  maximum  of  a 
given  indicator  was  only  half  as  Important  to  the 
effectiveness  of  the  unit  as  the  most  important 
maximum,  it  was  given  a  value  of  +50.  An  analogous 
process  was  then  completed  for  the  minimum  vaiues  of 
each  indicator.  Since,  by  going  through  this 
procedure,  each  indicator  for  a  given  KRA  had  been 
weighted  according  to  its  relative  value  or  Importance 
as  a  measure  of  Its  contribution  to  the  effectiveness 
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of  the  unit,  the  values  of  each  Indicator  were  able  to 
be  mathematically  summed  to  yield  the  total  effect¬ 
iveness  contribution  for  each  KRA, 

Since  It  Is  probable  that  all  KRAs  do  not  con¬ 
tribute  equally  to  mission  effectiveness,  each  KRA  had 
to  be  weighted  using  a  procedure  similar  to  the  pro¬ 
cedure  used  to  weight  the  Indicators  for  the  KRAs. 

This  was  necessary  since  weighting  the  KRA  scores 
allowed  the  Individual  weighted  KRA  scores  to  be 
aggregated  In  a  manner  similar  to  the  weighted  In¬ 
dicator  scores  to  produce  a  single  number  to  represent 
total  mission  effectiveness. 

The  facilitator  then  scheduled  a  review  of  the 
KRAs,  Indicators  and  their  ME  Charts  with  the  division 
officer,  as  was  done  In  the  study  by  Tuttle  and  Weaver 
< 1986a,  1986b).  In  reviewing  the  KRAs,  Indicators  and 
their  related  ME  Charts,  the  division  officer  had  sev¬ 
eral  options:  He  could  delete  the  item,  modify  the 
Item,  retain  the  Item  or  defer  action  on  the  Item  until 
further  research  was  done  to  determine  the  feasibility 
of  using  It.  The  facilitator  then  prepared  a  report  of 
the  results.  A  sample  ME  Chart  and  Indications  of  how 
ME  Charts  can  be  used  by  management  are  shown  In 
Figures  3  through  5. 
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Figure  3  presents  the  general  form  of  an  ME  Chart. 
The  horizontal  axis  represents  the  amount  of  an  In¬ 
dicator  ranging  from  the  possible  worst  to  the  possible 
best  level  of  performance.  The  vertical  axis  of  the 
figure,  which  ranges  from  a  maximum  effectiveness  of 
+100  to  a  minimum  effectiveness  of  -100,  Illustrates 
the  effectiveness  values  of  the  various  levels  of  the 
Indicator.  The  zero  point  or  zone  on  the  vertical  axis 
Is  defined  as  the  expected,  or  neutral,  level  of 
effect l veness. 

As  indicated  In  Figure  4,  the  MDT  considered  be¬ 
tween  8  and  11  repaired  Items  returned  per  month  the 
expected  level  of  performance.  Once  the  zero  or 
expected  level  of  performance  had  been  establ  Ished  by 
the  MDT,  the  next  task  was  to  determine  the  possible 
best  and  worst  levels  of  performance  to  “ anchor M  the 
values  on  the  graph. 

Figure  4  shows  that  the  MDT  Indicated  that.  In 
their  view.  It  Is  not  possible  to  do  better  than  5 
Y-Codes  crepaired  Items  returned)  per  month  and  they 
consider  20  repaired  l terns  returned  the  worst  possible 
level  of  performance.  From  Figure  4,  It  can  also  be 
ooserved  that  personnel  in  the  MDT  considered  the 
Y-Code  (return  rate)  an  Important  aspect  of  their  job. 
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9lnce  the  maximum  possible  effectiveness  is  *100  and 
Che  minimum  possible  effectiveness  Is  -90,  Although 
the  MDT  considered  a  return  rate  of  between  8  and  11 
Items  per  month  an  expected  level  of  performance.  It 
can  be  observed  that  the  decrease  In  effectiveness  Is 
not  linear  as  the  number  of  returned  Items  per  month 
Increases  beyond  11. 
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INDICATOR 
Figure  3 

General  form  of  a  Mission  Effectiveness  (HE)  Chart 


NUMBER  OF  Y— COOC  OCCURRENCES 


Figure  4 

How  to  interpret  the  KB  Chart 
Y-Code  Rate 
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U1S30N  EFFECT! Vtr£SS 


PBRCINT  OF  WORK  COTTER  PRODUCTION  EFFORT 


Figure  5 

Dl££erent-3hape<l  KB  Charts 
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Figure  S  (continued) 
Different *  Shaped  KB  Charts 
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Four  representative  ME  Charts  are  presented  In 
Figure  5.  For  each  Indicator,  there  Is  an  ME  Chart 
with  Its  possible  maximum,  possible  minimum,  expected 
level  or  zero  point,  and  a  function  relating  to 
effectiveness.  The  first  ME  Chart  represents  percent 
of  Work  Center  Production  Effort.  Note  that,  for  this 
ME  Chart,  the  expected  level  of  performance  Is  between 
92  and  100  percent  production  effort.  If  the  percent 
of  Work  Center  Production  Effort  Increases  to  110 
percent,  the  maximum  effectiveness  of  +90  Is  attained. 
Once  maximum  effectiveness  Is  attained  at  110  percent, 
further  Increases  In  effectiveness  are  not  posslDle. 
The  minimum  level  of  effectiveness  of  -05  Is  not 
achieved  until  percent  Work  Center  Production  Effort 
decreases  to  80  percent:  however,  the  function 
describing  the  drop  from  the  expected  level  of  Work 
Center  Production  Effort  to  the  minimum  possible  level 
of  Work  Center  Production  Effort  is  not  linear. 

The  second  ME  Chart  describes  the  percent  of 
completed  Personnel  Qualification  Standards  (PQS). 

Note  that  the  expected  level  of  performance  is  41 
percent  of  the  PQS  completed.  The  maximum  level  of 
effectiveness  of  +100  is  not  reached  until  the  percent 
of  completed  PQS  Is  Increased  to  55  percent,  ana  the 
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minimum  level  of  effectiveness  of  -90  Is  not  reached 
until  the  percent  of  PQS  has  decreased  to  20  percent. 
Note  also  that  there  Is  a  linear  relationship  through¬ 
out  the  range  of  completed  PQS  as  would  be  expected, 
since  there  is  no  specific  level  of  training  given  that 
would  produce  a  disproportionate  increase  or  decrease 
in  effectiveness  of  the  work  center  personnel.  With 
that  in  mind,  management  would  be  aware  of  the  fact 
that  effectiveness  gained  through  this  type  of  training 
yields  a  linear  return  throughout  Its  range. 

The  third  indicator,  monthly  Ready  For  Issue  (RFI) 
Rate,  has  an  expected  range  of  between  80  and  85  per¬ 
cent.  The  maximum  level  of  effectiveness  of  +100  Is 
not  reached  until  the  RFI  Rate  has  increased  to  95 

percent.  The  minimum  level  of  effectiveness  of  -90  Is 

not  reached  until  the  RFI  Rate  has  decreased  to  60 

percent:  however,  once  the  RFI  Rate  has  decreased  from 

80  to  75  percent,  the  majority  of  the  effectiveness  has 
been  lost.  This  Information  allows  managers  to  see 
that  the  largest  gains  In  effectiveness  can  be  made  in 
this  zone,  since  once  effectiveness  has  dropped  to  75 
percent,  little  more  can  be  lost  beyond  that  point. 

The  last  indicator,  number  of  monthly  Safety 
Violations,  has  an  expected  value  of  10  per  month.  The 
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maximum  effectiveness  of  +100  Is  reached  when  the 
number  of  Safety  Violations  has  decreased  to  one  per 
month,  and  the  minimum  effectiveness  of  -50  Is  reached 
when  the  number  of  Safety  Violations  has  increased  to 
30  per  month.  Note  especially  that  there  Is  a  non¬ 
linearity  between  number  of  Safety  Violations  per  month 
and  effectiveness,  and  that  the  greatest  Increases  In 
effectiveness  per  unit  of  change  can  be  achieved  as  the 
number  of  Safety  Violations  decreases  from  12  to  6. 

Two  things  are  particularly  noteworthy  about  the 
four  ME  Charts  shown  In  Figure  5.  The  first  Is  that 
the  ME  Charts  permit  non- 1  1  near 1 1 1 es  to  exist  between 
performance  on  Indicators  and  mission  effectiveness.  A 
linear  relationship  means  that  to  Improve  a  given 
amount  at  the  low  end  of  an  Indicator  yields  an  In¬ 
crease  In  effectiveness  as  great  as  Improving  that  same 
amount  at  the  high  end  or  anywhere  else  on  the  X-axis. 
In  the  work  environment,  this  Is  not  always  the  case, 
since  It  Is  very  common  for  values  in  the  mid-range  ci 
an  Indicator  to  represent  large  Improvements  in  pro¬ 
ductivity  and  the  values  at  the  high  end  Cor  low  end) 
to  represent  a  point  of  diminishing  returns.  When  an 
organization  achieves  a  fairly  high  level  of  product- 
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ivlty  In  one  given  Indicator,  It  might  prove  more 
productive  to  try  to  Improve  on  another  indicator  that 
is  not  doing  well,  rather  than  to  expend  the  extra 
effort  to  continue  improvement  on  an  indicator  that  is 
already  at  a  high  level. 

The  second  item  worthy  of  note  is  that  the  slope 
of  the  function  expresses  the  relative  Importance  of 
the  Indicator.  For  example,  the  overall  slope  of  the 
third  indicator  (RFI  Rate)  Is  steeper  than  the  fourth 
indicator  (Safety  Violations).  This  reflects  the  fact 
that,  although  it  Is  important  not  to  have  any  safety 
violations  while  working  on  an  item,  it  Is  more  im¬ 
portant  to  actually  complete  the  repair  and  have  the 
item  Ready  For  Issue  (to  the  customer). 

It  is  necessary  to  recognize  what  these  two 
properties  of  an  ME  Chart  can  reveal  to  the  manager. 
First,  since  the  slopes  can  be  non-i  inear,  they 
accurately  reflect  the  realities  of  how  an  organization 
functions.  In  many  cases,  linear  relationships  between 
Indicators  and  mission  effectiveness  do  not  exist. 
Secondly,  the  charts  reflect  different  levels  of 
Importance  for  different  indicators,  since  all  in¬ 
dicators  are  not  equally  important  to  the  productivity 
of  an  organization.  Aspects  of  the  job  that  are  very 
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important  get  steeper  slopes  than  aspects  of  a  job  that 
are  less  important:  therefore,  a  small  increase  in  a 
critical  aspect  of  a  Job  could  realistically  produce  a 
larger  Increase  in  effectiveness  than  a  larger  Increase 
in  a  non-crltical  aspect  of  the  job.  Thus,  the  re¬ 
lative  importance  of  each  aspect  of  the  Job  is  incor¬ 
porated  during  the  development  of  the  ME  Charts. 

Since  monthly  feedback  of  performance  on  the  ME 
Charts  was  necessary,  the  Data  Analysis  Division  of  the 
AIMD  assisted  in  generating  a  program  that  would  pro¬ 
vide  a  monthly  computer-printout  to  reflect  mission 
effectiveness  for  both  managers  and  workers.  The 
computer-generated  feedback  system  that  was  created  was 
able  to  provide  feedback  in  a  graphical  form,  show  the 
current  status  of  each  indicator,  and  aggregate  all 
effectiveness  information  to  a  single  measure  of 
productivity  for  the  AIMD  Officer. 

To  aggregate  effectiveness  scores,  it  was  nec¬ 
essary  to  weight  the  summed  effectiveness  scores  to 
reflect  the  relative  importance  of  the  KRAs.  This  was 
accomplished  by  asking  the  division  officer  and  the 
petty  officer  supervisors  in  the  MDT  to  assigned  ranks 
and  percentage-of- 100  weights  to  the  four  KRAs.  These 
were  40  percent  for  customer  satisfaction,  20  percent 
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for  motivation.  30  percent  for  training,  and  10  percent 
for  safety.  To  create  the  monthly  aggregated  score  for 
the  feedback,  a  weighted  mean  was  computed  with  the 
following  equation:  KRA  #1  summed  effectiveness  score 
(KRA  #1  weight)  +  KRA  #2  summed  effectiveness  score 

(KRA  #2  weight)  ♦  KRA  #3  summed  effectiveness  score 

(KRA  #3  weight)  +  KRA  #4  summed  effectiveness  score 

(KRA  #4  weight ) . 

An  alternate  to  this  double-weighting  procedure  is 
to  ask  the  MDT  to  evaluate  the  possible  worsts  and 
possible  bests  of  each  indicator  relative  to  all  the 
others  in  terms  of  Its  contribution  to  the  overall 
mission.  To  not  evaluate  possible  worsts  and  bests 
within  each  KRA  permits  the  effectiveness  scores  to  be 
sunuied  across  all  KRAs  and  avoid  the  second  weighting 
for  KRA  Importance.  The  problem  with  the  single 
weighting,  however,  is  that  the  MDT  may  resist  or  find 
it  too  difficult  to  evaluate  a  large  number  of  possible 
worsts/bests.  This  was  the  case  in  this  field  test. 

Once  the  computer-generated  reports  were  com¬ 
pleted,  the  Implementation  Phase  of  the  MGEEM  process 
began.  The  KRAs.  indicators,  and  ME  Charts  were 
published  and  distributed  to  all  personnel  In  the 
branch.  Management  support  was  evident  and  all  key 
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persons  showed  strong  support  for  the  system.  The 
Initial  monthly  ME  Charts*  reflecting  changes  In 
effectiveness  In  the  branch  during  the  Baseline  and 
Development  Phase,  were  distributed  to  both  management 
and  workers.  The  results  were  posted,  as  shown  In 
Appendix  G,  to  ensure  they  could  be  viewed  frequently. 

During  the  first  enhancement  of  monthly  feedback, 
the  researcher  did  not  Interact  with  the  target  organ¬ 
ization,  since  Interaction  with  branch  personnel  could 
have  possibly  caused  Increases  In  organizational 
productivity  due  to  the  workers/  knowing  that  they  were 
being  observed.  During  the  second  enhancement  of 
monthly  feedback  and  goal -sett Ing,  the  researcher 
Interacted  with  the  MDT  personnel  for  on  1  y  a  few  hours 
to  assist  them  In  setting  meaningful  goals.  A  sample 
of  two  ME  Charts  showing  mission  effectiveness  is  shown 
in  Appendix  H.  Feedback  was  Intended  to  motivate  the 
target  organization's  personnel  to  Increase  their 
product i v i ty . 


Analysis  of  Data 

A  description  of  the  analysis  of  data  is  presented 
in  this  section.  The  process  Included  listing  KRAs, 
charting  the  monthly  values  of  the  Indicators  on  ME 
Charts,  and  producing  a  graph  showing  overal 1  mission 
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effectiveness.  The  ME  Charts  displayed  a  baseline  of 
historical  data  that  were  collected  to  show  a  standard 
of  mission  effectiveness  from  which  changes  In  product¬ 
ivity  could  be  calculated.  The  charts  of  monthly 
values  on  the  Indicators,  Individual  ME  Charts,  and  the 
aggregate  productivity  graph  were  capable  of  displaying 
data  for  a  report  of  findings. 

Included  In  this  collection  of  data  for  analysis 
was  an  overlay  of  overall  aggregated  productivity  data 
on  a  single  graphic  display,  which  was  able  to  show  the 
combined  changes  In  mission  effectiveness  during  the 
terra  of  the  research  on  both  the  experimental  and 
control  facility.  Comparisons  of  the  graphic  display 
contributed  to  answering  the  research  question  posed  by 
the  study. 

Statistical  data  obtained  from  the  monthly  me¬ 
asurements  of  effectiveness  indicators  for  each  KRA 
provided  the  basic  Input  for  each  ME  Chart,  which 
indicated  monthly  performance  in  a  graphical  format. 
Numerical  comparisons  were  made  between  the  effect¬ 
iveness  scores  of  the  previous  month's  productivity 
data . 
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The  data  collected  from  the  Air  Force  study  at 
Bergstrom  Air  Force  Base  are  also  listed  for  comparison 
with  the  data  collected  during  this  study.  The  com¬ 
parisons  between  KRAs,  Indicators  and  resultant  pro¬ 
ductivity  changes  at  both  facilities  were  made  to 
determine  the  feasibility  of  lnter-servlce  transport¬ 
ability  of  the  MGEEM. 

Research  Aaaumpt loaa. and  JcLLmltflUcas 

The  following  research  assumptions  and  delimit¬ 
ations  are  discussed.  I  terns  not  listed  as  assumptions 
or  delimitations  were  not  considered  relevant  to  this 
study . 

Research  Assumptions 

In  designing  the  study,  it  was  assumed  that  the 
adoption  of  a  descriptive-correlational  research  ap¬ 
proach  using  Mission  Effectiveness  (ME)  Charts  and  an 
overall  mission  effectiveness  graph  along  with  tables 
1 l sting  KRAs.  indicators  and  changes  in  mission 
effectiveness  would  facilitate  in  the  collection  and 
analysis  of  the  data  required  to  answer  the  research 
questions.  It  was  *lso  assumed  that  the  research 
findings  that  were  limited  to  the  Communication  and 
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Navigation  Equipment  Repair  Branch  might  be  more 
broadly  appl ied  to  the  entire  AIMD. 

As  Is  the  case  with  any  performance  improvement 
strategy,  there  were  potential  dangers  and  problems 
associated  with  the  process.  Among  the  problems,  three 
fears  were  referred  to  by  Tuttle  and  Weaver  (1986b). 

One  fear  Is  the  fear  of  being  measured.  Since  concerns 
are  often  expressed  when  performance  is  measured,  these 
concerns  had  to  be  addressed.  The  second  fear,  the 
fear  of  failure,  was  dealt  with  by  encouraging  groups 
to  set  goals  that  were  challenging  but  attainable.  The 
third  fear,  the  fear  of  repercussions  from  higher  auth¬ 
ority,  was  frequently  grounded  in  a  history  of  having 
been  confronted  by  a  superior  for  a  performance 
problem,  but  this  fear  was  minimized  by  discouraging 
micromanagement.  That  was  accomplished  by  sending  only 
aggregated  data  to  the  AIMD  officer. 

Goals  can  be  seen  as  ceil  ings.  but  the  goals  used 
in  this  study  were  designed  to  serve  as  targets  that 
encouraged  performance  improvement.  The  key  to  avoid¬ 
ing  ceiling  effects  was  to  establish  in  the  organ¬ 
ization  attainable  goals  and  a  mind-set  of  continual 
work  on  performance  improvement. 
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One  normally  obtains  improvements  In  the  results 
that  one  measures;  therefore,  the  MDT  tried  to  consider 
only  the  important  facets  of  the  organizat Ion's  per¬ 
formance.  The  NGT  assisted  In  this  process,  since  it 
necessarily  reduced  the  KRAs  and  indicators  to  a  small 
number,  which  eliminated  the  measurement  of  less 
Important  results. 

PellnlUtlQna 

The  research  concentrated  on  identification  of 
KRAs  and  indicators  combined  with  measuring  the  overall 
effectiveness  of  the  Communication  and  Navigation 
Equipment  Repair  Branch  of  major  West  Coast  shore-based 
Naval  AIMDs  under  the  Commander,  Naval  Air  Force, 

United  States  Pacific  Fleet.  The  specific  KRAs 
measured  did  not  represent  a  universal  measure  of  total 
productivity,  but  were  Intended  to  be  representative  of 
areas  in  which  it  was  most  desirable  that  mission 
effectiveness  be  measured.  The  non-critical  areas  of 
productivity  were  eliminated  In  the  voting  process  used 
by  the  MDT. 

The  curves  representing  mission  effectiveness  were 
generated  by  consensus  of  the  MDT.  As  branch  prior¬ 
ities  and  MDT  membership  changed,  the  curves  on  the 
Mission  Effectivenss  Charts  were  held  constant  so  that 
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the  study  would  be  consistent  with  the  Air  Force  study. 
That  consistency  was  required  to  satisfy  the  restrict¬ 
ions  of  Research  Question  2.  In  the  real  world, 
managers  would  want  to  change  the  slopes  of  the  ME 
curves  to  reflect  different  priorities  as  the  environ¬ 
ment  changes. 

Summary 

The  Methodology  for  Generating  Efficiency  and 
Effectiveness  Measures  CMGEEM)  Is  a  multi-step  process 
that  enables  any  organization  to  create  a  complete 
productivity  measurement  and  enhancement  system.  The 
use  of  a  feedback  procedure  consisting  of  Mission 
Effectiveness  CMS)  Charts  allows  the  organization  to 
Incorporate  command  policy  and  goals  Into  the  mea¬ 
surement  process  by  constructing  possibly  non-linear 
feedback  charts  used  In  conjunction  with  the  product¬ 
ivity  measurement  system  that  has  been  created 
specifically  for  that  organization.  The  total 
productivity  measurement  system  so  created  contains 
significant  motivational  properties,  since  It  has  been 
created  by  the  workers  themselves  to  measure  their  own 
productivity.  It  also  provides  motivation,  since  It 
gives  the  workers  an  opportunity  to  discuss  their 
constraints  as  a  group,  formulate  solutions  to  those 
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constraints,  and  attempt  to  work  smarter,  not  harder. 
Once  the  worker-created  total  productivity  measurement 
system  had  been  designed,  feedback  alone  and  then 
feedback  coupled  with  goal -set ting  were  used  to 
determine  the  effects  on  mission  effectiveness. 

This  chapter  described  the  participative  product¬ 
ivity  measurement  and  Improvement  process  called  the 
MGEEM.  It  described  the  method  used  for  the  collection 
of  data,  outlined  the  data  sources,  and  described  the 
Instrumentation  design  that  was  used  as  a  feedback 
mechanism  to  managers  and  served  also  to  answer  the 
research  questions.  The  procedure  used  to  collect  the 
data  was  then  described,  along  with  specific 
preliminary  steps  that  were  used  to  facilitate  the 
Introduction  and  acceptance  of  the  MGEEM  process  at  the 
site  being  tested.  The  discussion  of  the  need  for 
advance  preparation  by  the  measurement  facilitator 
demonstrated  how  acceptance  of  the  MGEEM  process  could 
oe  either  hindered  or  facilitated.  The  Instrument  used 
to  collect  the  data  was  then  discussed.  The  analyses 
of  the  data  Lncluded  calculating  changes  In  mission 
effectiveness  and  examining  the  similarities  between 
the  productivity  measurement  and  enhancement  system 
designed  at  Naval  Air  Station  North  Island  and  at  an 
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Air  Force  base  located  in  the  southwestern  United 
States.  The  assumptions  and  delimitations  of  the  study 
were  listed. 


00 


Chapter  A 


FINDINGS 

This  chapter  presents  the  findings  of  the  study. 
The  first  part  of  the  study  consisted  of  identifying 
Key  Result  Areas  CKRAa)  and  indicators,  and  const¬ 
ructing  Mission  Effectiveness  (ME)  Charts  to  record 
performance  on  each  indicator.  The  study  then  obtained 
statistical  data  related  to  mission  effectiveness, 
hence  productivity,  using  four  months  of  historical 
data  to  establish  a  baseline,  four  months  of  current 
data  during  development,  four  months  of  data  while 
feedback  alone  was  used,  and  four  months  of  data  while 
both  feedback  and  goa  1 -sett  i ng  were  used.  The  second 
part  of  the  study  used  the  KRAs,  indicators  and 
resultant  changes  In  effectiveness  produced  in  the 
first  part  of  the  study  and  compared  them  with  the 
findings  of  a  similar  study  of  an  Air  Force  facility 
having  the  same  operational  responsibilities.  Section 
One  of  this  chapter  deals  with  data  relating  to  the 
first  research  question,  and  Section  Two  deals  with 
data  relating  to  the  second  research  question. 
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First  Research  Question 


What  changes  In  effectiveness  of  the  Communication 
and  Navigation  Equipment  Repair  Branch  of  a  shore-baaed 
Naval  Aircraft  Intermediate  Maintenance  Department 
CAIMD)  were  observed  during  and  after  implementing  the 
Methodology  for  Generating  Efficiency  and  Effectiveness 
Measures  <MGEEM)? 

These  results  will  be  presented  in  three  sections: 
<1)  measures  of  organizational  productivity  recorded 
from  historical  data,  which  constituted  the  baseline 
period  and  four  month's  current  data  for  the  period 
when  the  system  was  under  development,  <2>  measures  of 
productivity  during  the  feedback  period  that  followed, 
but  before  the  goal -setting  period  began,  and  (3) 
measures  of  productivity  after  goal -sett  1 ng  began . 

Results  During. the  Baseline 

and  S y s t cm_ Dev e lopmen t 

A  Measurement  Development  Team  <MDT>  consisting  of 
the  Division  Officer.  Work  Center  Supervisor  and  eight 
other  work  center  personnel  was  used  In  identifying  the 
various  KRAs.  indicators  and  constructing  ME  Charts. 
While  developing  the  KRAs.  a  total  of  36  potential  KRAs 
were  initially  listed  using  the  data-gather 1 ng  phase  of 
the  Nominal  Group  Technique  <NGT>.  After  discussion. 
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clarification,  modification  and  consolidation  of  the  36 
Initially-listed  KRAs,  7  remained,  and  these  were  then 
voted  on  to  further  reduce  the  list  and  identify 
priorities.  The  seven  candidate  KRAs  are  listed  in 
Appendix  F. 

Each  of  the  10  members  of  the  MDT  selected  5  of 
the  7  remaining  KRAs*  which  were  considered  most  Im¬ 
portant.  and  ranked  them  In  order  of  their  perceived 
Importance  <5  being  the  most  important).  Total  scores 
were  then  calculated  to  determine  the  items  that  were 
considered  the  most  Important  measurable  aspects  of 
performance  or  KRAs.  For  example,  i f  9  of  the  members 
of  the  MDT  each  assigned  a  rank  of  4  to  represent  the 
relative  Importance  of  a  specific  candidate  KRA .  then 
that  KRA  would  be  assigned  a  total  score  of  9  X  4  or 
36.  The  KRAs  having  the  highest  total  scores  from  the 
rankings  and  the  highest  "N.a  or  number  of  the  MDT 
members  voting  for  them,  were  considered  to  be  the  most 
Important  measurable  aspects  of  the  organization's 
mission:  the  remaining  KRAs,  having  received  a 
significant  drop  in  total  score  in  the  voting,  were 
eliminated  by  consensus  of  the  MDT.  Table  5  lists  the 
final  voting  score  of  KRAs  that  were  selected. 
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Table  5 


Final  Voting  Score  on 
Key  Result  Areas 
CN»  10  Members  of  the  MDT) 

Kev: 

Total  Score  =  Number  points  received  (i-5  points  per  vote) 
Scale:  1  =  least  Important:  moat  Important 


Key  Result  Area 

N 

Total  Score 

Customer  Satisfaction 

10 

50 

Mot  1  vat  Ion 

10 

33 

Train  1 ng 

10 

27 

Safety 

9 

24 

Tables  6  through  9  list  the  Indicators  that  were 
determined  by  the  NGT  to  be  the  factors  that  would  give 
the  best  evidence  that  the  specific  KRA  was  being  ac¬ 
complished.  Each  KRA  was  considered  separately  and,  using 
the  NGT  referred  to  earlier,  an  initial  set  of  indicators 
for  that  specific  KRA  was  listed.  Those  initial  candidate 
indicators  were  then  discussed,  modified  and  consolidated 
to  reduce  the  total  number  for  the  specific  KRA.  The 
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reduced  list  of  possible  Indicators  was  then  voted  on  by 
the  10  members  of  the  MDT.  They  ranked  the  resulting 
seven  Indicators  In  terms  of  Importance  <7=  most  Im¬ 
portant).  The  results  were  totaled,  where  N  represents 
the  number  of  members  of  the  MDT  who  considered  the 
Indicator  Important  and  total  voting  score  represents  the 
total  number  of  points  each  Indicator  received.  Only  the 
Indicators  with  the  highest  N  and  total  voting  score 
(prior  to  a  significant  decline  In  N  and  total  voting 
score)  were  selected  as  Indicators;  the  lower-scoring 
Indicators  were  el Iminated  by  consensus  of  the  MDT. 
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Table  6 


Indicators  Chosen  to  Measure  the  KRA 
Customer  Satisfaction 


Key.- 

Total  Score  =  Number  points  received  (1-7  points  per  vote) 
Scale:  l  =  least  Important:  7=  most  Important 


Indicator 

N 

Total  Score 

Ready  For  Issue  rate 

10 

63 

Turn  Around  Time 

9 

51 

Y-Code  rate 

9 

39 

Exrep  backlog  Items 

7 

34 

Backlog 

9 

28 

Broad  Arrow/TED  reports 

7 

27 

Table  7 

Indicators  Chosen  to  Measure  the  KRA 
Mot l vat  Ion 


Kev : 

Total  Score  =  Number  points  received  <1-7  points  per  vote) 
Scale :l  =  least  Important:  7  =  most  Important 


Indicator 

N 

Total  Score 

Recogn i 1 1  on 

9 

42 

Retention  percentage 

10 

40 

Negative  personnel  indicators 

8 

35 

Morale  Index 

7 

34 

Production  effort 

6 

32 
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Table  8 


Indicators  Chosen  to  Measure  the  KRA 
Training 

Kev : 

Total  Score  =  Number  points  received  (1-7  points  per  vote; 
Scale:  1  =  least  Important;  7*  most  important 


Indicator  N  Total  Score 

On-the-Job  Training  9  41 
Personnel  Qualification  Standards  9  39 
Rate  training  7  35 
Formal  billet  training  7  34 


Table  9 

Indicators  Chosen  to  Measure  the  KRA 
Safety 


Total  Score  -  Number  points  received  <1-7  points  per  vote; 
Scale: 1  =  least  Important:  7=  most  important 


Indicator 

N 

Total  Score 

Accidents 

9 

55 

Safety  viol  at  ions 

7 

38 

Repeat  safety  discrepancies 

8 

30 

07 


After  Identification  of  the  moat  Important  KRAs  and 
Indicators,  the  ME  Charts  were  were  developed  for  each 
Indicator:  these  are  shown  In  F! cures  6  through  23.  To 
develop  each  ME  Chart,  group  discussion  was  used  to  de¬ 
termine  several  points  before  smooth  lines  were  used  to 
connect  the  points.  The  first  point  that  was  determined 
was  the  zero  point  or  “neutral1*  zone,  which  represents  an 
expected  level  of  performance.  This  level  of  performance 
was  seen  as  neither  causing  management  to  commit  more 
resources  for  increasing  productivity  nor  causing  manage¬ 
ment  to  praise  the  group  for  Its  performance.  Once  the 
expected  level  or  zone  of  performance  was  determined, 
doth  the  best  and  worst  possible  levels  of  performance 
envisioned  by  the  MDT  for  the  branch  were  ascertained. 
This  "anchored*  the  extreme  points  of  the  horizontal  axis 
of  the  ME  Chart.  Intermediate  points  of  performance  were 
then  determined  by  discussion,  and  a  line  was  then  drawn 
connecting  those  points. 
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Figure  6 

Mission  Effectiveness  Chart  for  the  Indicator 
Heady  For  Issue  (RFI)  Hate 


Figure  7 

Mission  Effectiveness  Chart  for  the  Indicator 
Turn-Around  Time 
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Figure  8 

Mission  effectiveness  Chart  for  the  Indicator 
Y-Code  Rate 


NUMBER  OF  EXREP  BACK  LOO  G  CD  ITEMS 


Figure  3 

Mission  effectiveness  Chart  for  the  indicator 
8xrep  Backlog 
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Mission  Effectiveness  Chart  for  the  Indicator 
Backlog 


Figure  XX 

Mission  Effectiveness  Chart  for  the  Indicator 
Broad  Arrow  /  TED  Reports 
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Figure  12 


Hlsslon  Effectiveness  chart  for  the  Indicator 
Recognition 


Figure  13 

Hlsslon  Effectiveness  Chart  for  the  Indicator 
Quarterly  Retention 
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MISSION  Q-FLCnvHNESS  MiSSON  EJFECnv**CSS 


Figure  14 

Mission  Effectiveness  Chart  for  the  Indicator 
Negative  Personnel  indicators 


Mission  Effectiveness  Chart  for  the  Indicator 
Morale  Index 
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percent  or  work  center  production  effort 


Figure  16 

Mission  Bffe ctiveness  Chart  for  the  Indicator 
Production  Effort 


Plgure  17 

Mission  Effectiveness  Chart  for  the  Indicator 
On-The-Job  (OJT)  Training 
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Figure  18 

Mission  Effectiveness  Chart  for  the  indicator 
Personnel  Qualification  Standards  (PQS) 


MONTHLY  RATE  TRAINING  WOURS /PERSON 


Figure  19 

Mission  Effectiveness  Chart  for  the  indicator 
Rate  Training 
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Figure  20 

Hiss  ion  Effectiveness  Chart  for  the  Indicator 
Fonal  Billet  Training 


Figure  21 

Mission  Effectiveness  Chart  for  the  indicator 
Accidents 
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NUMBER  or  SAFETY  VIOLATIONS 


Figure  22 

Hiss  Ion  Effectiveness  Chart  for  the  indicator 
Safety  Violations 


Figure  23 

Hlsslon  Effectiveness  Chart  for  the  indicator 
Repeat  Discrepancies 
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After  the  entire  framework  of  the  productivity 
measurement  and  enhancement  system  had  been  developed, 
the  Indicators  were  again  considered  by  the  MDT.  During 
this  discussion,  the  Indicators  were  weighted  based  on 
their  perceived  Importance  as  measures  of  mission  effect¬ 
iveness.  The  indicator  selected  by  the  MDT  to  have  the 
highest  level  of  importance  for  Its  contribution  as  a 
measure  of  productivity  for  the  specific  KRA  was  auto¬ 
matically  given  the  value  of  MOO,  the  highest  possible 
effectiveness  value.  The  effectiveness  values  were 
assigned  In  Increments  of  five  units  to  more  easily 
enable  the  MDT  to  reach  a  consensus  on  a  value.  The  MDT 
then  rated  the  other  Indicators  relative  to  the  most 
Important  indicator.  For  example.  If  by  consensus  of  the 
MDT  the  maximum  for  a  given  indicator  wa3  only  considered 
half  as  Important  to  the  effectiveness  of  the  unit  as  the 
most  Important  maximum.  It  was  given  a  value  of  +50. 

This  procedure  weighted  the  Indicators  of  each  KRA 
relative  to  each  other  as  measures  of  effectiveness. 

Table  10  lists  the  Indicators  for  each  JCRA  and  their 
respective  weights. 
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Tabl e  10 


Relative  Weights  of  the  Various 
Indicators  as  Measures  of 
Mission  Effectiveness 


ICRA  *  Indicator  **  Weighted  Score 


Customer 

Ready  For  Issue  rate 

100 

Satl  sf act  ion 

Y-Code  rate 

100 

Exrep  backlog  Items 

95 

Backlog 

90 

Broad  Arrow/TED  reports 

85 

Turn  Around  Time 

sa 

530  Total 

Mot  l  vat  Ion 

Recogn l t Ion 

100 

Morale  Index 

95 

Production  effort 

90 

Retention  percentage 

Negat l ve  personne l 

40 

1 ndlcators 

85 

410  Total 

Tral n  ing 

On-the-JoD  Training 

100 

Rate  training 

90 

Formal  billet  training 
Personnel  Qualification 

80 

Standards 

100 

370  Total 

Safety 

Accidents 

100 

Safety  violations 

100 

Repeat  safety  discrepancies 

250  Total 

#  Detailed  descriptions  of  KRAs  may  De  seen  In  Table  3 
(Chapter  3) . 

##  Detailed  descriptions  of  Indicators  may  De  seen  in  Table 
4  (Chapter  3) . 
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Once  each  of  the  indicators  was  weighted,  the  KRAs 
had  to  be  weighted  in  a  manner  slml  iar  to  that  used  to 
weight  the  indicators,  so  that  a  single  number  repre¬ 
senting  overall  productivity  could  be  generated.  The 
relative  weights  of  each  KRA  were  used  in  the  formula: 
KRA  #1  summed  effectiveness  score  (KRA  #1  weight)  +  KRA 
#2  summed  effectiveness  score  (KRA  #2  weight)  ♦  KRA  #3 
summed  effectiveness  score  (KRA  #3  weight)  +  KRA  #4 
summed  effectiveness  score  (KRA  #4  weight).  The 
weights  are  listed  in  Table  11. 

Table  11 

Relative  Weights  of  the  KRAs  as  Measures 
of  Mission  Effectiveness 


KRA 

Maximum  Score  X  Weight 

=  Weighted 
Score 

Customer  Satisfaction 

530 

X 

.40 

= 

212 

Mot i vat  ion 

410 

X 

.20 

82 

Trai n i na 

370 

X 

.30 

= 

ill 

Safety 

250 

X 

.10 

= 

25 

Total 

430 

100 


Once  the  KRAs  and  Indicators  had  been  Identified  and 
weighted  and  ME  Charts  had  been  constructed,  a  computer 
feedback  program  was  designed  that  would  Incorporate  the  KRA 
weighting  formula  and  print  out  Individual  graphs  for  each 
KRA  and  a  composite  graph  for  overall  productivity.  While 
the  computer  feedback  program  was  being  developed,  four 
months  of  baseline  data  were  gathered  from  historical  files, 
and  four  months  of  current  data  were  collected  for  both  the 
experimental  and  control  facilities  at  North  Island  and 
Cemoore ,  California,  respectively.  Those  data  points  are 
presented  In  Tables  12  through  15. 
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Table  12 


Naval  Air  Station  North  Island,  Coronado,  California 
Effectiveness  During  the  Four-Month 
Base l Ine  Per  1 od 
(Experimental  Facl l 1 ty) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  Indicates  performance  better 
than  level  set  by  Measurement  Development  Team, 


1987  Baseline 


KRA  # 

Jan 

Feb 

Mar 

Apr 

Customer  Satisfaction 

-189.0 

-  71.0 

-165,0 

-  95.0 

Mot l vat  Ion 

-  73.2 

-  15.2 

21.8 

26.8 

Training 

90.0 

95.0 

-  12.0 

-  27.0 

Safety 

250.0 

250.0 

198.0 

250.0 

Tota 1  Effect  1  veness 

77.8 

258.8 

42.8 

154.8 

*  Detailed  descriptions  of  KRAs  can  be  found  In  Table  3 
< Chapter  3) . 


After  a  baseline  had  been  established  using  four  months 
of  historical  data,  four  months  of  current  data  were 
collected  during  the  Development  Phase. 
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Table  13 


Naval  Air  Station  North  Island,  Coronado,  California 
Effectiveness  During  the  Four-Month 
Development  Period 
(Experimental  Faci 1 i ty ) 

Scale:  Higher  values  indicate  greater  effectiveness. 

Any  value  above  zero  Indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


KRA  * 

May 

1987  Development 

Jun  Ju 1 

Aug 

Customer  Satisfaction 

-116.0 

49.0 

123.0 

146.0 

Mot  1  vat  ion 

-  81 .2 

52.8 

31 .0 

70  .8 

Training 

90.8 

-  13.3 

-  49.6 

-  14.0 

Safety 

250.0 

115.0 

243.0 

250  .0 

Total  Effectiveness 

143.6 

203.5 

347.4 

452.8 

#  Detailed  descriptions  of  KRAs  can  be  found  in  Table  3 
(Chapter  3). 


At  the  Control  Facility  In  Lemoore .  California,  a 
Dasellne  was  also  established  using  four  months  of 
historical  data. 
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Table  14 


Naval  Air  Station  Lemoore,  Lemoore,  California 
Effectiveness  During  the  Four-Month 
Basel  1 ne  Period 
(Control  Facl 1 1 ty ) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  Indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


KRA  * 

Jan 

1907 

Feb 

Base  1 1 ne 

Mar 

Apr 

Customer  Satisfaction 

135.0 

31 .3 

19.3 

103.0 

Motivation 

93.2 

-  54.2 

-  24.2 

-  54.2 

Training 

61.3 

289.0 

370.0 

209.0 

Safety 

26.0 

250.0 

120.0 

242.5 

Total  Effect  Iveness 

77.1 

516.  1 

405.1 

500.3 

#  Detailed  descriptions  of  KRAs  can  be  found  in  Table  3 
(Chapter  3) . 


After  a  baseline  had  been  establ lshed  using  four  months 
of  historical  data,  four  months  of  current  data  were 
collected  during  the  Development  Phase. 
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Table  15 


Naval  Air  Station  Lemoore,  Leraoore ,  California 
Effectiveness  During  the  Four-Month 
Development  Period 
(Contro 1  Faci 1 l ty ) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  Indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


KRA  * 

May 

1907 

Jun 

Devel opment 

Jul 

Aug 

Customer  Sat  lsfact  Ion 

273.7 

351.5 

206.3 

143.5 

Mot  1  vat  Ion 

-54.2 

-  54.2 

-  54.2 

23.8 

Tral  n  1  ng 

347.9 

289.0 

156.4 

370.0 

Safety 

250 .0 

250.0 

250  .0 

226.0 

Total  Effectiveness 

817.4 

836.3 

558.5 

763.3 

#  Detailed  descriptions  of 

KPAs  can  be 

found  In  Table 

3 

(Chapter  3). 

During  the  Development  Phase,  a  Morale  Index 
survey  was  administered  quarterly  to  the  branch 
personnel  by  the  branch  supervisor  at  each  location. 
The  Morale  Index  survey  Is  shown  in  Appendix  G.  Its 
variables  were  Jod  Satisfaction.  Morale.  Reenlistment 
Intentions.  Role  Clarity  and  Clarity  of  Objectives. 
These  variables  were  scored  on  five-point  Likert 
scales.  The  scores  were  added  to  produce  a  single 
number  to  be  used  as  a  Morale  Index,  and  then  the 
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number  was  entered  as  an  Indicator  called  Morale  Index 
in  the  KRA  Motivation. 

Once  the  productivity  measurement  and  enhancement 
system  was  constructed  and  a  baseline  of  four  months  of 
historical  data  plus  four  months  of  development  data 
were  plotted,  it  was  calculated  that  overall  average 
productivity  at  Naval  Air  Station  North  Isiand  in¬ 
creased  14.6  percent  over  baseline  during  the  Develop¬ 
ment  Phase,  as  shown  in  Figure  24. 

Resu-Ua-Hurlng.  the  Ecur-Montli 

FceasasK  <;  Enhancement  12 

During  the  Feedback  Phase,  the  researcher  de¬ 
liberately  avoided  i nteract i ng  wi th  the  target 
organization.  This  was  for  the  purpose  of  eliminating 
any  possible  Increases  in  organizational  productivity 
due  to  the  workers'  knowing  that  they  were  being 
observed.  Feedback  graphs,  however,  were  placed  In 
high  traffic  areas  and  were  seen  by  all  work  center 
personnel.  The  results  of  the  data  collection  during 
the  Feedback  Phase  are  listed  in  Tables  16  ana  17  for 
the  experimental  and  control  facilities,  respectively. 
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Table  16 


Naval  Air  Station  North  Island,  Coronado,  California 
Effectiveness  During  the  Four-Month 
Feedback  (Enhancement  I) 

(Experimental  Facl 1 1 ty) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  Indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


1987  Feedback 


KRA  * 

Sept 

Oct 

Nov 

Dec 

Customer  Satisfaction 

191.3 

46.4 

142.6 

16.7 

Mot  1  vat  Ion 

-108.3 

30.9 

190.6 

189.4 

Training 

-  34.9 

96.5 

23.2 

90.5 

Safety 

250.0 

120.0 

250.0 

250.0 

Total  Effectiveness 

298.1 

191  .0 

614.4 

546.6 

#  Detailed  descriptions  of  KRAs 

can  be 

found  1  n 

Table  3 

(Chapter  3) . 


At  the  Control  Facility  In  Lemoore,  California,  four 
months  of  data  were  recorded  during  the  Feedback  Phase. 
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Table  17 


Naval  Air  Station  Lemoore.  Lemoore.  California 
Effectiveness  During  the  Four-Month 
Feedback  <  Enhancemen  t  I ) 

(Control  Faci l  i  ty ) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


KRA  # 

1907 

Sept 

Feedback 

Oct 

Nov 

Dec 

Customer  Satisfaction 

-  27.7 

161.3 

101.8 

70.0 

Motivation 

-171  .2 

57.8 

86.0 

23.8 

Training 

370.0 

370.0 

370.0 

370.0 

Safety 

244.0 

120.0 

238.0 

250.0 

Total  Effectiveness 

415. i 

709.1 

796.6 

713.8 

*  Detailed  descriptions  of  KRAs  can  be  found  in  Table  3 
(Chapter  3). 

Results  During  the  Four -Months Feedback 

and  Goal -Setting  (Enhancement  II) 

During  the  Feedback  and  Goal -Setting  Phase,  the 
researcher  i n teracted  wi th  the  Measurement  Development  Team 
(MDT)  approximately  one  hour  per  month  to  set  goals  ana 
discuss  the  previous  month's  performance  data  for  each  of 
the  indicators  In  each  KRA .  Roadblocks  that  stood  in  the 
way  of  increasing  mission  effectiveness  during  the  previous 
month  and  anticipated  roadblocks  for  the  coming  month  were 
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discussed.  These  discussions  assisted  the  Measurement 
Development  Team  In  setting  more  realistic  goals  and  In 
overcoming  or  finding  ways  around  occasional  roadblocks. 
The  data  collected  for  the  experimental  facility  and  the 
control  facility  are  presented  in  Tables  i8  and  19, 
respect  1 ve 1 y . 
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Table  18 


Naval  Air  Station  North  Island  Effectiveness 
During  the  Four-Month  Feedback  and 
Goal-Setting  (Enhancement  II) 
(Experimental  Facl 1 l ty ) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  indicates  performance  better 
than  the  level  set  by  Measurement  D 
evelopment  Team. 


1988  Feedback  and  Goal -Setting 


KRA  » 

Jan 

Feb 

Mar 

Apr 

Customer  Satisfaction 

86.3 

37.3 

54 . 4 

187.3 

Mot i vat  ion 

183.7 

103.4 

244.4 

120.4 

Trai ning 

122.6 

290.0 

370.0 

370.0 

Safety 

250.0 

243.0 

250.0 

115.0 

Total  Effect  1  veness 

642.6 

673.7 

918.8 

792.7 

*  Detailed  descriptions 

i  of  KRAs 

can  be 

found  In  Table 

3 

(Chapter  3) . 

At  the  Control  Facl 

1 Ity  in 

Lemoore , 

,  Cal l f orni a .  four 

months  of  aata  were  recorded  during  the  Goal-Setting  Phase. 


no 


Table  19 


Naval  Air  Station  Lemoore  Effectiveness 
During  the  Four-Month  Feedback  and 
Goal -Setting  (Enhancement  II) 

(Contro I  Faci I  1 ty ) 

Scale:  Higher  values  Indicate  greater  effectiveness. 

Any  value  above  zero  indicates  performance  better 
than  the  level  set  by  Measurement  Development  Team. 


KRA  # 

1988  Feedback 
Jan 

and  Goal 
Feb 

-Sett ing 
Mar 

Apr 

Customer  Satisfaction 

282.4 

138.0 

249.5 

227.2 

Mot  1  vat i on 

69.8 

86.3 

69.8 

-  23.2 

Training 

370.0 

370.0 

370.0 

370.0 

Safety 

120.0 

120.0 

250.0 

250.0 

Total  Effectiveness 

842.2 

714.8 

939.3 

824.0 

*  Detailed  descriptions  of  KRAs  can  be  found  in  Table  3 
(Chapter  3). 


After  applying  the  proper  weights  to  each  KRA's  summed 
effectiveness  score,  the  weighted  mean  effectiveness  scores 
were  determined  for  each  four-month  phase  of  the  experiment. 
Those  weighted  mean  effectiveness  scores  are  listed  in  Taole 
20. 
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Table  20 


Weighted  Mean  Effectiveness  Scores 
for  each  Four-Month  Phase 
of  the  Experiment 


Phase 

Weighted  Mean 

Effectiveness  Score 

Base  1 i ne 

-  19.3 

Deve 1 opment 

46.4 

Feedback 

90.1 

Goal-Setting 

177.0 

During  feedback,  average  mission  effectiveness  in¬ 
creased  22.1  percent  over  baseline.  In  calculating  per¬ 
cent  increase  in  effectiveness,  the  ratio  of  the  gain  in 
effectiveness  to  the  maximum  possible  gain  in  effect¬ 
iveness  was  used.  The  gain  in  effectiveness  is  cal¬ 
culated  by  subtracting  the  mean  baseline  effectiveness 
(-19.3)  from  the  mean  effectiveness  during  the  given 
period:  the  maximum  possible  gain  in  effectiveness  is 
calculated  by  subtracting  the  mean  baseline  effectiveness 
from  the  maximum  possible  effectiveness  (430).  For 
example,  since  the  mean  effectiveness  during  feedback  was 
80.1.  that  was  a  gain  of  99.4  over  mean  Oaseiine 
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effectiveness  <80 . l-<- 19.3) ) .  The  percent  gain  In 
effectiveness  of  the  maximum  possible  gain  In  effective¬ 
ness  is  99.4/  449.3,  or  22.1  percent.  Other  Increases  in 
mission  effectiveness  were  calculated  In  a  similar 
manner.  This  approach  to  calculating  change  is.  In  some 
ways,  conservative.  In  the  sense  that  the  maximum 
possible  Increase  In  effectiveness  Is  limited  to  100 
percent . 

During  goa 1 -sett l ng,  average  mission  effectiveness 
increased  43.7  percent  over  baseline,  as  shown  in  the 
graph  in  Figure  24.  The  overall  changes  In  mission 
effectiveness  during  the  term  of  the  study  for  the  con¬ 
trol  facility  at  Lemoore  Is  shown  in  Figure  25.  An 
overlay  showing  the  comparison  of  mission  effectiveness 
during  the  12  months  of  data  collected  In  the  study 
between  the  experimental  and  control  facility  i3  shown  in 
Figure  26.  A  Trend  Line  comparing  mission  effectiveness 
of  the  experimental  and  control  facility  during  the  same 
12-month  period  is  shown  in  Figure  27. 
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Figure  26 

Twelve-Month  Comparison  of  Mission  Effectiveness 
of  Naval  Air  Stations  North  Island  and  Lemoore 
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Figure  27 

Twelve-Month  Trend  Line  Comparison  of  Mission  Effectiveness 
of  Naval  Air  Stations  North  Island  and  Lemoore 
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Sg.coad,  Research  J3uaaLLan 

How  did  the  KRAs,  Indicators  and  the  subsequent 
changes  In  effectiveness  obtained  from  this  Navy  study 
compare  with  those  of  an  Independently-conducted  Im¬ 
plementation  In  an  Air  Force  shop  with  an  Identical 
organizational  mission? 

The  KRAs  and  Indicators  from  Tables  5  through  9  have 
been  consolidated  by  branch  of  service  and  are  presented 
In  Tables  21  and  22  to  facilitate  the  comparison. 

Table  21 

Final  Key  Result  Areas  and  Indicators 
for  this  Study  Done  in  the 
United  States  Navy 


M  —  Cuatomgr__Sat  Isf action 

Y-Code  rate;  Percentage  of  repaired  equipment  that  did 
not  function  immediately  after  installation. 

Backlog:  Number  of  Items  that  were  awaiting  maintenance 
(AWM)  or  awaiting  parts  (AWP). 

Exrep  backlog  Items:  Number  of  Items  that  were  AWM  or 
AWP  for  expeditious  repair. 

RFI  rate:  Percentage  of  equipment  brought  in  for  repair 
that  was  actually  repaired. 

Turn  Around  Time:  Average  amount  of  time  taken  to 
repair  items  In  the  repair  process. 

Broad  Arrow/TED  reports:  Number  of  test  benches  that 
were  Inoperative  or  in  a  degradea  condition. 
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Table  21  Ccontlnued) 


KBA  J  2.  .  .Mot  Iv&tlQn 

Recognition:  Number  of  people  formally  recognized  or 
rewarded  per  month. 

Moral  Index:  Average  of  feedback  scores  on  a  quarterly 
morale  survey. 

Production  effort:  Percentage  of  work  accomplished 
by  work  center  personnel. 

Negative  personnel  indicators:  Number  of  negative 
reports  received  on  assigned  personnel . 

Quarterly  Retention:  Percent  of  those  eligible  who 
reenlist  or  extend  on  active  duty. 

KRA  — i— 2a—  Xr*iRl.n.g 

OJT:  Average  number  of  On-The-Job  Training  hours 
documented  per  month  per  person. 

PQS:  Percent  of  assigned  Personnel  Qualifications 
Standards  that  have  been  completed. 

Rate  training:  Average  hourly  amount  of  training 
completed  In  Navy  professional  jobs. 

Formal  billet  training:  Average  hourly  amount  of 

training  received  In  specific  job  assignment. 

KRA  Ji, _ 2 AffiJUL 

Accidents:  Number  of  reported  accidents  at  work. 

Safety  Violations:  Number  of  safety  violations 
reported  at  work. 

Repeat  Discrepancies:  Number  of  repeat  discrepancies 
on  the  Safety  Audit. 
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The  KRAa  and  Indicators  determined  by  the  Air  Force 


study  are  listed  In  Table  22. 


Table  22 

Final  Key  Result  Areas  and  Indicators 
for  the  Previous  Study  Done  in  the 
United  States  Air  Force 


ZRh _ ft  1-u^quipmca^  KePdic 

Bounces:  Percentage  of  repaired  equipment  that 
did  not  function  Immediately  after 
instal 1  at  ion  . 

Percent  QA  (Quality  Assurance)  Inspections  passed: 
Rate  of  QA  Inspections  that  were  passed. 

AWM:  Number  of  units  awaiting  maintenance. 

AWP:  Number  of  units  awaiting  parts. 

Demand  Met:  Percentage  of  equipment  brought  In 
for  repair  that  was  actually  repaired. 

KRb _ ft  2.  Training 

STS  Tasks  Completed:  Mean  number  of  standard 
(more  basic)  training  tasks  completed  for 
personnel  In  training. 

Percent  Quai  Tasks  Completed,  Comm:  Mean  percent  of 
advanced  training  tasks  completed  for  personnel 
repairing  communications  equipment. 

Percent  Qua!  Tasks  Completed,  NAV :  Mean  percent  of 
advanced  training  tasks  completed  for  personnel 
repairing  navigation  equipment. 

Scheduled  Training  Tasks  Overdue:  Total  nuraoer 
non-technical  (e.g.,  military)  training 
requirements  not  met  on  time  for  all  shop 
personnel . 
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Table  22  (continued) 


& BA- 1.3 ,  _  Q.thcr  JhiUea 

Mobility  Equipment:  Number  of  pieces  of  equipment 
used  for  mobility  exercises  that  were  not 
calibrated  by  the  shop  on  schedule. 

PMEL  Overdue:  Number  of  pieces  of  shop 

calibration  and  test  equipment  that  were  not 
calibrated  by  the  shop  on  schedule. 

Percent  349  Errors:  Percent  of  errors  on  a  major 
manpower  documentation  form. 

Missed  Appointments:  Number  of  formal  on-base 
appointments  missed. 


The  percent  increases  in  mission  effectiveness  for 
the  Air  Force  base  during  the  feedback  and  feedback  with 
goal-setting  periods  of  the  study  are  compared  vl  th  those 
obtained  from  the  Navy  study  in  Table  23.  The  main  ob¬ 
servation  is  that  effectiveness  Increased  in  both  organ¬ 
izations  during  feedback  and  further  Increased  in  both 
organizations  during  feedback  with  goa l-set t l ng.  The 
differences  between  the  increases  in  the  two  organ¬ 
izations  may  be  explained,  at  least  in  small  part,  by  the 
fact  that  the  Air  Force  facility  figures  are  based  on  a 
five-month  feedback  and  a  five-month  goal -setting  period, 
while  the  Navy  facility  figures  are  based  on  a  four-month 
feedback  and  a  four-month  goal -setting  period.  More 
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Importantly,  the  Navy  facility  underwent  a  major  command 
Inspection  during  the  second  month  of  the  Feedback  Phase. 
During  that  month,  overall  effectiveness  decreased  to 
deiow  average  Development  Phase  effectiveness. 

Table  23 

Mission  Effectiveness  Increases  of 
Navy  and  Air  Force  Facilities 
During  MGEEM 


Military  Branch  Percent  Increases  Percent  Increases 

of  Service  During  Feedback  Feedback 

&  Goal -Set ting 


Navy  22.1*  43.7* 

Air  Force  30.0*  65.0* 


Summary 

This  chapter  presented  the  findings  of  the  study. 
The  results  of  analyses  conducted  to  answer  the 
research  questions  were  presented  in  individual  and 
composite  formats  to  aid  in  visual  analysis.  The  first 
section  of  the  chapter  listed  the  changes  In  effective¬ 
ness  of  two  Communication  and  Navigation  Equipment 
Repair  Branches  of  shore-based  Naval  Aircraft 
Intermediate  Maintenance  Departments  CAIMDs)  that  were 
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documented  during  and  after  the  use  of  the  Methodology 
for  Generating  Efficiency  and  Effectiveness  Measures 
(MGEEM).  The  second  section  of  the  chapter  reported 
findings  pertaining  to  Key  Result  Areas,  Indicators  and 
subsequent  changes  in  mission  effectiveness  obtained  In 
the  Navy  study  and  compared  them  with  the  results  from 
an  Independently-conducted  implementation  In  a  shop 
with  the  same  organizational  mission  In  the  Air  Force. 
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Chapter  5 

SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 

The  Information  Included  In  this  chapter  sum¬ 
marizes  the  findings  that  resulted  from  the  Implement¬ 
ation  of  the  research  design  specified  for  the  study. 
Also  Included  are  conclusions  drawn  from  data  gathered 
during  the  study.  The  chapter  concludes  with 
reconimendat Ions  for  further  research  related  to  the 
study . 


Sumnar.y 

The  following  paragraphs  discuss  the  research 
problem,  delineate  the  research  population  and  specify 
the  theoretical  framework  of  the  study.  The  background 
leading  up  to  the  study  and  the  method  of  analysis 
employed  In  the  study  are  also  outlined. 

The  Research  Problem 

Studies  of  methods  to  Increase  mission  effective¬ 
ness  emphasize  either  feedback  and  goal -setting  or 
using  worker  participation  In  designing  measurement 
criteria.  The  Improvements  resulting  from  combining 
these  two  approaches,  when  applied  In  the  military 
services  of  the  United  States,  have  been  supported  and 
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demonstrated  in  studies  and  reports  by  the  Air  Force 
Human  Resources  Laboratory  CAFHRL)  (Pritchard,  Jones 
and  Roth  1987:  Tuttle  and  Weaver  1986a:  1986b). 

Because  of  the  Navy's  traditional  use  of  strict 
chal n-of -command  and  strict  Junior-senior  relation¬ 
ships,  worker  participation  in  management  decisions  has 
historically  received  little  attention.  In  the  modern 
environment  of  decreasing  resources,  increasing  costs 
and  increasing  complexity  of  weapons  systems,  a 
reconsideration  of  command  relationships  and  conditions 
required  investigation  if  productivity  targets  estab¬ 
lished  by  Executive  Order  12552  were  to  be  realized  by 
1992. 

This  study  examined  the  use  of  a  participative 
management  approach  to  measure  and  enhance  mission 
effectiveness  in  a  Navy  facility  called  the  Methodology 
for  Generating  Efficiency  and  Effectiveness  Measures 
(MGEEM).  The  MGEEM  approach  to  measuring  and  enhancing 
productivity  was  field-tested  in  numerous  Air  Force 
settings  (Pritchard,  Jones  and  Roth  1987:  Tuttle  and 
Weaver  1986a:  1986b)  and  was  found  to  increase  mission 
effectiveness,  to  be  well-liked  by  workers,  and  to  be  a 
cost-effective  method  for  measuring  and  enhancing 
productivity.  The  changes  in  mission  effectiveness  of 
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two  Navy  facilities,  one  an  experimental  facility  and 
one  a  control  facility,  were  monitored  during  and  after 
implementation  of  the  MGEEM  in  the  experimental 
facility.  Additionally,  the  Key  Result  Areas  (KRAs) 
and  indicators  generated  in  the  Development  Phase  of 
the  MGEEM  and  the  subsequent  changes  in  mission  effect¬ 
iveness  observed  in  this  Navy  study  were  compared  with 
those  from  a  simiilar  implementation  conducted  earlier 
by  the  AFHRL  in  an  Air  Force  Communication  and 
Navigation  Equipment  Repair  Shop  with  an  Identical 
organizational  mission. 

Two  major  objectives  were  specified  for  this 
study.  The  first  was  to  document  the  changes  in 
mission  effectiveness  of  the  Communication  and  Nav¬ 
igation  Equipment  Repair  Branch  of  a  shore-based  Naval 
Aircraft  Intermediate  Maintenance  Department  ( AIMD) 
during  and  after  implementation  of  the  MGEEM.  The 
second  objective  was  to  determine  the  inter-service 
transportability  of  the  MGEEM  by  observing  how  closely 
the  KRAs.  Indicators  and  subsequent  changes  in  mission 
effectiveness  obtained  in  this  Navy  study  compared  with 
those  that  resulted  from  an  independent  l  y-conducteo 
implementation  in  a  simiilar  organization  in  the  Air 
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Force.  In  view  of  these  objectives,  the  study  was 
designed  to  answer  the  following  questions: 

1.  What  changes  In  effectiveness  of  the  Comm¬ 
unication  and  Navigation  Equipment  Repair  Branch  of  a 
shore-based  Naval  Aircraft  Intermediate  Maintenance 
Department  CAIMD)  were  observed  during  and  after 
Implementing  the  Methodology  for  Generating  Efficiency 
and  Effectlvenes  Measures  <MGEEM>? 

2.  How  closely  did  the  Key  Result  Areas  (KRAs), 
Indicators  and  subsequent  changes  In  effectiveness 
oDtalned  from  this  Navy  study  compare  with  those  of  an 
independently-conducted  Implementation  In  an  Air  Force 
shop  with  an  Identical  organizational  mission? 

Del ineatlon 

This  study  focused  on  a  system  designed  to  measure 
and  enhance  mission  effectiveness  and  Investigated  the 
transportability  of  that  system.  Only  shore-based 
Naval  AIMDs  were  studied  while  attempting  to  enhance 
mission  effectiveness  In  the  Navy;  however,  the  results 
of  an  Air  Force  productivity  enhancement  effort  In  a 
shop  with  an  Identical  organizational  mission  were  used 
to  investigate  the  transportatol  1  l  ty  of  the  system.  It 
was  proposed  in  this  study  that  changes  in  mission 
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effectiveness  occur  when  interventions  of  feedback  and 
goal-setting  are  used  In  conjunction  with  a  worker¬ 
generated  measurement  system.  It  was  also  proposed 
that  inter-service  transportability  of  the  system  might 
be  feasible.  The  system  under  study  for  measuring  and 
enhancing  productivity  emphasized  the  Initial  use  of 
worker  and  supervisor  participation  In  generating  the 
Initial  measurement  system  itself.  Subsequent  inter¬ 
ventions  of  feedback  and  goal-setting  were  then  applied 
to  investigate  their  effects  on  overall  mission 
effectiveness.  The  system  requires  participative 
management  during  construction  of  measurement  para¬ 
meters,  and  It  then  emphasizes  the  use  of  feedback 
alone  as  an  Intervention  and  still  later  emphasizes  the 
use  of  both  feedback  and  goal-setting  to  increase 
mission  effectiveness. 

Theoretical  Framework 

The  theoretical  framework  of  this  study  was  drawn 
from  the  Air  Force  Human  Resources  Laboratory  (AFHRL) 
reports  on  the  use  of  the  Methodology  for  Generating 
Efficiency  and  Effectiveness  Measures  (MGEEM)  by 
Pritchard.  Jones  and  Roth  (1987)  and  Tuttle  and  Weaver 
(1986a:  1986b).  Integration  of  the  results  of  the  two 

reports  supports  the  use  of  active  participation  of 
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workers  and  supervisors  when  designing  a  productivity 
measurement  and  enhancement  system.  Interventions  of 
feedback  alone  and  then  goal-setting  coupled  with 
feedback  were  investigated  to  determine  their  Influence 
on  Increasing  mission  effectiveness. 

Background 

Productivity  was  among  the  most  important  issues 
facing  the  nation:  however,  this  area  had  not  been 
adequately  addressed  in  the  United  States  Navy.  Since 
the  AFHRL  had  conducted  several  studies  and  produced 
numerous  reports  on  productivity  measurement  and 
enhancement  projects  in  the  Air  Force,  it  followed  that 
the  methods  that  were  successful  for  the  Air  Force 
should  be  examined  in  the  Navy. 

The  Communication  and  Navigation  Equipment  Repair 
Branches  of  Naval  AIMDs  were  used  for  this  Navy  study 
since  they  have  the  same  operational  mission  as  an  Air 
Force  Communication  and  Navigation  Equipment  Repair 
Shop  that  recently  was  used  to  field  test  the  MGEEM 
methodology.  By  examining  the  resultant  KRAs.  indi¬ 
cators  and  the  corresponding  changes  in  mission 
effectiveness  in  this  Navy  study,  the  inter-service 
transportability  of  the  system  could  be  tested 
simu 1 taneous 1 y . 
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In  view  of  the  need  for  the  development  of  an 
accurate  method  for  measuring  and  enhancing  product¬ 
ivity  In  a  Navy  setting  and  considering  that  no 
published  study  had  yet  focused  on  measuring  and 
enhancing  productivity  In  a  Naval  AIMD,  this  study 
filled  a  need. 

Method 

The  sample  for  this  study  was  composed  of  the 
Communication  and  Navigation  Equipment  Repair  Branches 
of  two  major  West  Coast  Naval  AIMDs.  Additionally,  the 
AFHRL  report  concerning  the  use  of  the  MGEEM  at  the 
Communication  and  Navigation  Equipment  Repair  Shop  at 
an  Air  Force  base  located  In  the  southwestern  United 
States  was  used  to  compare  results  of  the  Navy  study 
for  determining  the  lnter-servlce  transportability  of 
the  MGEEM  system. 

This  sample  represented  two  of  the  seven  major 
West  Coast  Naval  AIMDs,  which  serve  numerous  squadrons 
that  have  various  types  of  aircraft  and  avionics 
equipment.  The  Conmun  i  cat  l on  and  Navigation  Equipment 
Repair  Branch  of  the  Navy  AIMD  at  Naval  Air  Station 
North  Island,  Coronado,  California  was  chosen  to 
implement  the  MGEEM  methodology,  while  the 
Communication  and  Navigation  Equipment  Repair  Shop  at 


129 


Naval  Air  Station  Lemoore,  Lemoore,  California  va3  used 
as  the  control  group. 

This  study  used  a  descriptive-correlational 
approach.  The  approach  was  considered  appropriate 
because  a  descriptive  procedure  was  needed  to  describe 
the  effects  of  feedback  alone  and  then  goal-setting 
coupled  with  feedback  on  productivity.  For  invest¬ 
igating  relationships  between  this  Navy  study  and  the 
Air  Force  study,  correlational  procedures  were  required 
to  determine  if  there  were  similarities  between  the 
KRAs,  indicators  and  resultant  changes  in  mission 
effectiveness  In  this  Navy  study  and  a  previously- 
conducted  Implementation  completed  by  the  AFHRL  in  an 
Air  Force  shop  with  an  identical  organizational 
mission.  The  comparison  would  help  in  determining  the 
feasibility  of  the  inter-service  transportability  of 
the  MGEEM. 

Conclusions 

Two  research  questions  were  formulated  for  the 
purpose  of  the  study.  The  major  findings  related  to 
these  two  questions  are  summarized  in  the  succeeding 
sect  ion . 
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Research.  Question  One 


What  changes  in  effectiveness  of  the  Communication 
and  Navigation  Equipment  Repair  Branch  of  a  shore-based 
Naval  AIMD  were  observed  during  and  after  Implementing 
the  MGEEM? 

One  of  the  effects  of  the  development  of  the  MGEEM 
system  was  that  It  produced  an  opportunity  for  the  or¬ 
ganization  to  carefully  examine  its  objectives,  pos¬ 
sible  measures  of  those  objectives,  and  obtain  a  true 
feeling  of  the  productivity  expectations  of  management. 
This  process  led  supervisors  to  examine  present  pro¬ 
cedures  and  discover  numerous  places  where  improvements 
could  be  made  in  the  overall  operation  of  the  organ¬ 
ization  during  the  Development  Phase  of  the  product¬ 
ivity  measurement  and  enhancement  system.  Naturally 
enough,  the  supervisors  began  to  Implement  these 
changes,  but  this  created  a  dilemma  for  the  researcher. 
While  it  was  clearly  worthwhile  for  the  organization  to 
improve  Its  effectiveness  during  the  development  of  the 
measurement  system,  this  improvement  occurred  prior  to 
the  Intervention  of  the  first  enhancement  of  feedback. 
If.  because  of  this,  the  Development  Phase  showed  a 
higher  effectiveness  than  it  otherwise  would  have,  this 
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would  tend  to  decrease  the  size  of  any  effect  due  to 
the  feedback  and  goal-setting  interventions. 

There  was  little  that  the  researcher  could  do 
about  this  dilemma.  The  supervisors  of  the  organ¬ 
ization  felt  strongly  that  such  changes  should  be  made 
and  they  made  them.  The  supervisors  also  felt  that 
these  changes  were  Increasing  their  effectiveness,  and 
this  Indeed  seemed  to  be  the  case,  since  mission 
effectiveness  increased  14.6  percent  over  baseline 
during  the  Development  Phase  of  MGEEM.  In  interviews 
with  the  supervisors,  it  was  ascertained  that  a 
substantial  portion  of  this  Improvement  was  due  to  the 
process  that  the  work  center  personnel  went  through 
while  developing  the  productivity  measurement  system. 
This  suggests  that  the  Improvements  In  productivity 
that  were  evidenced  in  the  later  Interventions  were,  i 
fact,  underestimates  of  the  overall  Impact  of  the 
enhancements  of  feedback  and  goal-setting  on  overall 
mission  effectiveness. 

Once  the  MGEEM  was  developed  and  a  baseline  of 
four  months  of  historical  data  were  plotted,  the 
results  Indicated  that  productivity  increased  22.1 
percent  over  baseline  when  feedback  alone  was  used 
during  the  Feedback  Phase,  as  Indicated  in  Figure  24. 
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As  the  figure  indicates,  overall  effectiveness 
continued  to  Increase  substantially  over  baseline. 
During  the  Feedback  and  Goal-Setting  Phase,  average 
productivity  increased  43.7  percent  over  baseline. 

These  percent  increases  are  percent  gain  In 
effectiveness  over  baseline  compared  with  the  maximum 
possible  gain. 

A  12-month  trend  line  shown  In  Figure  27  (Chapter 
3)  compares  the  changes  in  mission  effectiveness  at  the 
experimental  facility  at  Naval  Air  Station  North 
Island.  Coronado.  Cal  i  f orni a  wl  th  those  of  the  control 
facility  at  Naval  Air  Station  Lemoore,  Lemoore, 
California.  From  the  trend  lines  it  can  be  observed 
that  the  overall  Increase  In  mission  effectiveness  at 
the  experimental  facility  was  clearly  greater  than 
changes  that  were  occurring  without  the  planned 
intervention  of  both  feedback  and  feedback  with 
goal-setting  activities  in  a  similar  organization. 
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Research  Quest i on _  Two 


How  closely  did  the  KRAs,  indicators  and 
subsequent  changes  In  effectiveness  obtained  from  this 
Navy  study  compare  with  those  of  an  i ndependent 1 y- 
conducted  implementation  in  an  Air  Force  shop  with  an 
identical  organizational  mission? 

The  number  of  KRAs  in  this  Navy  study  was  four: 
the  Air  Force  only  used  three.  Since  “Safety"  was 
considered  the  least  important  KRA  in  the  Navy  study, 
and  since  a  paral lei  ism  must  be  established  with  the 
Air  Force  study  to  facilitate  making  a  comparison,  the 
KRA  “Safety*  Cwith  its  corresponding  Indicators)  was 
eliminated  from  the  analysis.  The  remaining  KRAs  are 
compared  in  Table  24,  and  it  can  be  observed  that  two 
of  the  three  KRA  areas  coincide.  The  Navy  KRA 
“Motivation0  did  not  appear  in  the  Air  Force  report  as 
a  KRA  since  the  Air  Force  did  not  consider  “Motivation" 
to  be  one  of  its  KRAs.  The  Navy  KRA  "Customer 
Satisfaction"  included  most  of  the  Indicators  used  in 
the  combined  Air  Force  KRAs  "Equipment  Repair"  and 
"Other  Duties,"  and  the  Navy  and  Air  Force  KRA 
"Training"  Included  highly  slml 1 lar  indicators. 
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Table  24 


Compari son 
The  Navy 

of  Key  Result  Areas  in 
and  Air  Force  Study 

NAVY 

AIR  FORCE 

Customer 

Sat isf act  ion 

Equipment  Repair 

Customer 

Sat isf act  ion 

Other  Duties 

Train ing 

Trai n 1 ng 

Mot l vat  1  on 

CAir  Force  did  not  choose 
as  a  KRAI 

The  Indicators  for  the  the  Navy  KRA  "Customer 
Satisfaction"  are  compared  to  the  parallel  Indicators  for 
the  Air  Force  KRAs  "Equipment  Repair"  and  "Other  Duties’*  in 
Table  25.  It  can  be  observed  that  five  of  the  six  Navy 
indicators  have  equivaient  Air  Force  indicators  for  the  KRAs 
under  inspection. 
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Table  25 


Comparison  of  Customer  Satisfaction  Indicators 
for  Navy  and  Air  Force  KRAs 
(  from  Tab  1 e  24) 


NAVY  KRA 

AIR  FORCE  KRA 

Customer  Satisfaction  Equipment  Repair  and 

Other  Duties 


Y-Code  Rate  (Repaired  equipment 
that  did  not  function  after 
insta 1 l at  ion ) 

Bounces  (Repaired  Items 
that  did  not  function 
after  instal 1  at  ion) 

Backlog  (Number  of  items  that 
were  Awaiting  Maintenance 
( AWM)  or  Awaiting  Parts 
<AWP)  for  repair) 

AWM  (Number  of  items 
Awaiting  Maintenance) 

AWP  (Number  of  items  that 
were  Awaiting  Parts) 

Exrep  Backlog  (Number  of  Items 
that  were  Awaiting  Maintenance 
(AWM)  or  Awaiting  Parts  (AWP) 
for  expeditious  repair) 

AWM  (Number  of  items 
Awaiting  Maintenance) 

AWP  (Number  of  items  that 
were  Awaiting  Parts) 

Ready  for  Issue  Rate 

(Equipment  brought  in  for 
repair  that  was  actually 
repaired) 

Demand  Met  (Equipment 
brought  in  for  repair 
that  was  actual  I  y 
repaired) 

Broad  Arrow/Test  Equipment 
Degradation  Reports  (Number 
of  test  benches  operating 
in  a  degraded  condition  or 
i noperable) 

PMEL  Overdue  (Shop 
cal  ibration  and  test 
equipment  working  in  a 
degraded  condition  due 
to  lack  of  calibration) 

Broad  Arrow/Test  Equipment 
Degradation  Reports  (Number 
of  test  benches  operating  in 
a  degraded  condition  or 
inoperable) 

Mobility  Equipment  (Pieces 
of  equipment  used  for 
mobility  exercises  in  a 
degraded  condition  due 
to  lack  of  calibration) 
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Table  25  (continued) 


NAVY  KRA 

Customer  Sat  Isfact Ion 


AIR  FORCE  KRAs 
Equipment  Repair  and 
Other  Duties 


Turn  Around  Time  (Average  time 
taken  to  repair  Items  In  the 
repair  process) 

CNavy  did  not  choose  as  an 
Indicator ] 


CNavy  did  not  choose  as  an 
Indicator ] 


CNavy  old  not  choose  as  an 
Indicator ] 


CAlr  Force  did  not  choose 
as  an  Indicator] 


Percent  Quality  Assurance 
Inspections  Passed  (Rate 
of  QA  Inspections  passed) 


Percent  349  Errors  (Rate 
of  errors  on  a  major 
manpower  documen tat l on 
form) 

Missed  Appointments 
(Number  of  formal 
on-base  appointments 
missed) 


The  Indicators  for  the  Navy  and  Air  Force  KRA 
"Training*1  are  compared  In  Table  26.  It  can  be  observed 
that  ail  four  of  the  Navy  Indicators  coincide  with  the  Air 
Force  Indicators  for  the  KRA  "Training.*  However,  the  Air 
Force  had  an  additional  Indicator  (Scheduled  training  tasks 
overdue)  that  the  Navy  did  not  use. 
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Table  26 


Comparision  of  Training  Indicators 
for  Navy  and  Air  Force  ICR  As 
(from  Table  24) 


NAVY  KRA  AIR  FORCE  KRA 

Training  Training 


On-the-Joti-Trainlng  (Basic 
training  hours  documented) 

Rate  Training  (Basic  Navy 
Job  training  completed) 

Personnel  Qualification 
Standards  (Advanced  job 
specific  tasks  completed) 

Formal  Billet  Training 
(Job  specific  training 
received  in  a  formal  school) 

CNavy  did  not  choose  as  an 
1 nd icator J 


STS  Tasks  Completed  (Basic 
training  tasks  comp l e ted) 

STS  Tasks  Completed  (Basic 
training  tasks  completed) 

Percent  Qual  Tasks  Complete 
Comm  &  Nav  (Advanced 
training  completed) 

Percent  Qual  Tasks  Complete 
Comm  &  Nav  (Advanced 
training  completed) 

Scheduled  Training  Tasks 
Overdue  (Number  of 
non-technical  training 
tasks  not  met  on  time) 


To  answer  the  third  part  of  Research  Question  Two, 
namely,  the  comparison  of  the  changes  in  productivity  of  the 
Navy  and  Air  Force  facillltles,  the  productivity  increases 
previously  documented  in  Research  Question  One  of  this  Navy 
Study  and  the  productivity  increases  previously  documented 
in  the  Air  Force  Study  are  listed  in  Table  27.  It  can  be 
observed  that  productivity  increased  significantly  during 
both  the  feedback  and  the  goal-setting  period  after 
implementation  of  the  MGEEM. 
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Table  27 


Increase  In  Mission  Effectiveness  of  Navy 
and  Air  Force  Facilities  During  MGEEM 
(Results  from  Research  Question  One 
and  the  Air  Force  Study) 


Branch  of  Service 

Mission  Effectiveness  Inccaasej. 

During  Feedback  Feedback  &  Goal -Setting 

Navy 

0'S 

CM 

43.7* 

Air  Force 

30  * 

65  * 

The  Increases  in  mission  effectiveness  recorded  In 
Table  27  present  strong  eviaence  to  substantiate  the  vla- 
blilty  of  using  the  MGEEM  system  to  increase  productivity  In 
the  Navy.  When  comparing  this  Navy  study  with  the  Air  Force 
study,  significant  similarities  In  KRAs  and  indicators  were 
observed:  however,  from  the  dissimilarities  observed.  It  is 
apparent  that  each  local  organization  has  to  tailor  the  set 
of  KRAs  and  Indicators  to  its  own  needs  prior  to  using  them 
to  gauge  mission  effectiveness.  The  transportability  of  the 
MGEEM  process  therefore  Is  accepted,  and  the  transport- 
anility  of  most  of  the  KRAs  and  their  indicators  Is  also 
evident,  although  parallelism  in  ail  details  has  not  been 
established.  Further  study  might  find  that  some  differences 
are  inevitable,  or  that  greater  parallelism  should  be 
sought . 
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Discussion  of  the  findings 


This  section  is  a  discussion  of  the  results  of  the 
analyses  conducted  to  answer  the  research  questions. 

1.  Discussion  of  Findings  of  Research  Question  One. 

It  was  found  that  there  was  a  significant  Increase 
in  overall  mission  effectiveness  at  the  Navy  facility 
that  Incorporated  the  MGEEM,  as  indicated  in  Figures 
24,  27,  and  28  (Chapter  3).  A  part  of  the  increase  was 
believed  to  be  due  to  either  the  Hawthorne  Effect 
(which  states  that  workers  tend  to  do  better  when  they 
know  they  are  being  observed)  or  the  documented  fact 
that  work  center  personnel  obtained  true  feelings  of 
the  productivity  expectations  of  management  during  the 
Development  Phase  and  made  changes  to  achieve  them 
before  enhancements  were  used.  This  was  apparent, 
since  average  mission  effectiveness  increased  14.6 
percent  over  baseline  during  the  Development  Phase  of 
MGEEM . 

During  the  Feedback  Phase  there  were  two  events 
that  occurred  that  may  have  served  to  decrease  overall 
mission  effectiveness.  A  major  command  Inspection  was 
held  in  Che  month  of  October,  and  extensive  prep- 
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aratlons  for  the  inspect  Iona  are  standard  operating 
procedure  in  the  Navy:  consequently,  the  decreases  in 
mission  effectiveness  for  both  September  and  October 
could  be  expected.  The  amount  of  preparation  and 
volume  of  non-productive  work  that  must  be  completed  to 
prepare  for  a  major  command  inspection  would  logically 
detract  from  the  productivity  of  the  organization  in 
the  month  of  the  Inspection  as  well  as  the  prior  month. 
The  other  such  event  during  feedback,  which  coincided 
with  a  decrease  in  effectiveness,  was  the  holiday 
vacation  period  in  December.  Historically,  military 
organizations  allow  up  to  50  percent  of  their  personnel 
to  go  on  leave  during  that  period.  Procuctivity 
naturally  would  decrease  when  a  large  part  of  the  work 
force  1 s  not  available  for  production. 

The  trend  lines  shown  in  Figure  27  (Chapter  3)  and 
the  composite  charts  showing  mission  effectiveness  for 
the  experimental  and  control  facilities  In  Figure  26 
show  clearly  that  a  far  greater  increase  in  overall 
productivity  occurred  In  the  Experimental  Facility, 
which  used  the  MGEEM  nethodoiogy.  Although  general¬ 
izations  from  a  single  experiment  of  this  type  are  not 
conclusive,  there  is  strong  evidence  to  Indicate  that 
use  of  the  productivity  measurement  and  enhancement 
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technology  that  is  incorporated  In  the  MGEEM  was 
responsible  for  a  significant  increase  in  overall 
mission  effectiveness  of  the  Communication  and 
Navigation  Equipment  Repair  Branch  of  the  Naval  AIMD. 

2.  Discussion  of  Findings  of  Research  Question  Two. 

It  was  found  that  the  Navy  used  four  KRAs,  while 
the  Air  Force  only  used  three  KRAs;  therefore,  before  a 
comparison  was  made,  the  least  important  Navy  KRA  and 
its  respective  indicators  were  not  considered  when  a 
comparison  between  the  studies  was  made.  Those  KRAs 
are  listed  in  Table  28. 


Tab l e  28 


Key  Result  Areas  used  in  The 
Navy  and  Air  Force  Study 


Navy 


Air  Force 


Customer  Satisfaction 


Equipment  Repair 


Training 


Training 


Mot i vat i on 


Other  Duties 


Safety 
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Of  che  13  Indicators  identified  in  the  Air  Force  study. 
9  indicators  from  the  Navy  study  were  found  to  coincide  witn 
them,  as  shown  in  Taole  29.  Since  66.6  percent  of  the  KRAs 
and  69.2  percent  of  the  Indicators  were  equivalent  tor  ooth 
service  organizations,  this  Indicates  that  there  would  oe  a 
significant  opportunity  to  provide  a  oaslc  framework:  of  the 
MGEEM-produced  KRAs  and  Indicators.  These  Dasic  KRAs  and 
indicators  could  oe  modified  locally  oy  the  organizations 
applying  them,  thus  permitting  the  organizations  to  allow 
for  regional  policy  and  operational  mission  alrferences. 

Table  29 

Comparisons  of  All  Indicators  from  tne 
Navy  and  Air  Force  Studies 


Navy 

Air  Force 

Y-Cooe  Race  (Repaired  equipment 
chat  did  not  function  after 
in3ta l  1  at i on) 

Backlog  (Number  of  items 
Awaiuna  Maintenance  (AWM) 
or  Awa  icing  Parts  ( AWP ) 
tor  repair) 

Exrep  Backlog  (Numoer  of  items 
Awa i t i ng  Ha  in tenance  (AWM) 
or  Awaiting  Parts  (AWP) 
for  expeditious  repair) 


Bounces  (Repaired  equipment 
that  did  not  work  after 
instal lat ion) 

AWM  (Number  of  items 
Awaiting  Maintenance) 

AWP  (Number  ot  items 
Awaiting  Parts) 

AWM  (Number  ot  Items 
Awaiting  Maintenance) 

AWP  (Number  of  items  tnat 
were  Awaiting  Parts) 


M3 


Table  29  (continued) 


NAVY 

AIR  FORCE 

Ready  for  Issue  Rate 

(Equipment  brought  in  for 
repair  that  was  fixed) 

Demand  Met  (Equipment 
brought  in  for  repair 
that  wa9  actually  fixed) 

Broad  Arrow/Test  Equipment 
Degradation  Reports  (Number 
of  test  benches  operating 
in  a  degraded  condition 
or  Inoperable) 

PMEL  Overdue  (ShoD 
calibration  and  test 
equipment  working  in  a 
degraded  condition  due  to 
lack  of  calibration) 

Broad  Arrow/Test  Equipment 
Degradation  Reports  (Number 
of  test  benches  operating 

In  a  degraded  condition 
or  Inoperable) 

Mobility  Equipment  (Pieces 
of  equipment  used  for 
mobility  exercises  in  a 
degraded  condition  due  to 
lack  of  calibration) 

On-the-Job  Training  (Basic 
training  hours  documented) 

STS  Tasks  Completed  (Basic 
training  tasks  completed) 

Rate  Training  (Basic  Navy 
job  training  completed.) 

STS  Tasks  Completed  (Basic 
training  tasks  completed) 

Personne 1  Qua  1 1 f l cat i on 
Standards  (Advanced  job 
specific  tasks  completed) 

Percent  Qual  Tasks  Complete 
Comm  8.  Nav  (Advanced 
training  completed) 

Formal  Billet  Training 
(Job  specific  training 

Percent  Qual  Tasks  Complete 
Comm  &  Nav  (Advanced 

received  in  a  formal  school)  training  complete) 

Turn  Around  Time  (Average  time  [Air  Force  did  not  choose  a 


taken  to  repair  Items  in 
the  repair  process) 

an  indicator] 

M4 


Table  29  (continued) 


NAVY 

AIR  FORCE 

CNavy  did  not  choose  as 
an  Indicator] 


CNavy  did  not  choose  as 
an  indicator] 


CNavy  did  not  choose  as 
an  indicator] 


CNavy  did  not  choose  as 
an  Indicator] 


Percent  Quality  Assurance 
Inspections  Passed  (Rate 
of  QA  inspections  that 
were  passed) 

Percent  349  Errors  (Rate  of 
errors  on  a  major  manpower 
documentation  form) 

Missed  Appointments  (Number 
of  formal  on-base 
appointments  missed) 

Scheduled  Training  Tasks 
Overdue  C Number  of 
non-technica l  training 
tasks  not  met  on  time) 


The  Increases  in  organizational  effectiveness  in  both 
the  Navy  and  Air  Force  studies  provide  strong  evidence  of  a 
significant  increase  in  productivity  associated  with  use  of 
the  MGEEM  in  both  the  Navy  and  the  Air  Force. 
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Ifflg-LigaUflua  .and  Regommcndatlgns 


A  number  of  Implications  and  recommendations  can 
be  drawn  from  this  study.  Several  relate  to  the  first 
research  question  concerning  the  viability  of  the 
MGEEM.  The  remainder  relate  to  the  second  research 
question  concerning  the  lnter-servlce  transportability 
of  the  MGEEM. 

IffiP-LlfiaUsag 

The  results  of  this  Navy  study  affirm  numerous 
earlier  studies  concerning  the  beneficial  effects  of 
feedback  and  goal -setting  on  organizational  product¬ 
ivity.  The  study  also  reaffirms  that  participation  of 
the  workers  in  designing  a  productivity  measurement 
system  motivates  the  work  force,  since  it  engenders  a 
sense  of  ownership  in  the  measurement  system. 

In  addition  to  significantly  increasing  product¬ 
ivity,  the  MGEEM  was  found  to  be  highly  acceptable  to 
both  management  and  workers.  Because  no  additional 
money,  materials,  facilities,  and  only  a  token  amount 
of  manpower  to  bul  Id  and  monitor  the  system  were 
required  to  realize  a  significant  increase  in  product¬ 
ivity.  the  MGEEM  proved  to  be  extremely  cost  effective. 
The  MGEEM  also  proved  to  be  relatively  easy  to 
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understand  and  Implement.  It  additionally  satisfied 
the  common  need  among  commanders  for  a  method  to  (U 
comprehensively  measure  organizational  productivity  and 
(2)  facilitate  the  manager's  Job,  since  Mission  Effect¬ 
iveness  Charts  allow  managers  to  see  monthly  where 
extra  emphasis  should  be  placed  to  increase 
product i vi ty . 

The  inter-service  transportability  of  the  basic 
MGEEM  system  was  also  supported.  However,  It  was  clear 
that  local  adjustments  to  a  basic  MGEEM  framework  need 
to  be  made  to  custom-fit  it  to  each  new  activity. 

A  number  of  questions  were  raised  as  a  result  of 
this  study.  Would  the  MGEEM  technology  be  appropriate 
In  a  wartime  environment?  To  state  this  question  dif¬ 
ferently,  participative  management  with  feedback  and 
goal -setting  serves  to  Increase  productivity  In  a 
peacetime  environment,  but  would  it  be  equally 
applicable  In  the  life  and  death  situations  of  a 
wartime  environment?  Would  the  strict  requirement  for 
chai n-of-command  and  junior-senior  relationships  have 
priority  over  participative  management?  How  would 
participative  management  work  in  other  Navy  settings? 
How  would  participative  management  work  in  other 
military  services? 


147 


Recommendatlona 


The  results  of  this  study  suggest  that  the 
Department  of  Defense  should  sponsor  a  full-scale  test 
of  the  Impact  of  using  this  measurement  and  enhancement 
technology  In  other  organizational  settings,  and  should 
test  the  transportability  of  the  MGEEM  for  use  In 
service  components  not  Included  In  this  study.  Sub¬ 
sequent  tests  of  the  MGEEM  should  consider  enhancements 
other  than  feedback  and  goal -setting  to  determine 
whether  the  participative  management  techniques  alone 
provide  Increases  in  mission  effectiveness.  Possibly, 
participative  management  techniques,  when  combined  with 
team  building  (Quality  Circles.  Juran  Teams,  etc.;  and 
other  enhancements,  would  provide  comparable  Increases 
in  mission  effectiveness. 

Further  research  should  be  conducted  to  determine 
whether  a  relationship  exists  between  MGEEM  and  the 
so-called  Deming  way  (Deralng,  1982;  i986).  Can  MGEEM 
be  the  implementation  technique  used  as  the  vehicle  to 
incorporate  Demlng's  "Fourteen  Points"  In  an  organ¬ 
ization?  How  could  this  be  Implemented?  For  example, 
do  the  measurement  system  construction  techniques  or 
the  feedback  sessions  used  In  the  MGEEM  give  workers  a 
chance  to  participate  In  two-way  communication  with 
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supervisors  to  a  great  enough  extent  to  “Drive  O^t 
Fear”?  Does  focusing  on  mission,  KRAs,  and  indicators 
achieve  a  measurable  outcome  consistent  with  Demlng's 
"Constancy  of  Purpose*?  What  are  other  ways  that  the 
MGEEM  could  Incorporate  Demlng's  philosophy  and  Ideas 
In  a  productivity  enhancement  setting? 

Suggestions  For. Further  Research 

The  findings  of  this  study  confirmed  the  existence 
of  an  increase  of  productivity  when  participative 
management  is  used  In  coordination  with  feedback  and 
goal -setting.  It  also  suggested  the  viability  of 
intra-service  transportability  of  the  basic  MGEEM. 

The  areas  In  which  further  study  would  be 
appropriate  Include: 

1.  Determining  whether  enhancements  other  than 
feedback  and  goal -set  ting  would  produce  slmlllar  or 
Detter  results. 

2.  Researching  the  possible  use  of  the  MGEEM 
technology  to  incorporate  Demlng's  "Fourteen  Points”  in 
an  organization. 

3.  Testing  on  a  larger  scale  the  Inter-service 
transportability  of  the  MGEEM  process. 
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4.  Finding  the  quickest  and  most  effective  way  to 
train  facilitators  and  to  reach  a  consensus  In  group 

si  tuat ions. 

5.  Investigating  the  best  types  of  automation  to 
use  for  processing  feedback  reports. 
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APPENDIX  A:  STRUCTURED  INTERVIEW 
Format  for  Interview  with  the  AIMD  Officer 


MQTELTO  INTERVIEWER 

Following  Introductions  and  preliminary  discussion  to 
establish  rapport,  the  following  questions  may  be  used  to 
gather  Information.  These  are  simply  opening  questions  and 
should  be  followed  by  "probes"  to  obtain  more  detail. 
Examples  of  probes  are:  "Tell  me  more  about  that."  "Could 
you  elaborate?"  "Could  you  give  me  a  specific  example?" 

The  purpose  of  the  Interview  Is  to  make  possible  the 
construction  of  a  systems  diagram  of  the  organization.  It 
is  helpful  to  visualize  this  diagram  as  the  Interview  is 
conducted. 


1.  What  is  the  mission  of  this  AIMD? 

2.  What  are  the  major  products/services  of  the  AIMD? 

3.  Who  are  the  principal  customers  of  the  organization? 

4.  What  about  staffing  levels?  How  many  military  and 
civilian  personnel  are  authorized?  Assigned?  What  Is 
the  breakdown  by  pay  grade  and  experience? 

5.  What  other  organizations  does  this  unit  depend  upon 
for  information  or  support  to  get  the  job  done? 

6.  What  degree  of  control  do  you.  as  AIMD  Officer,  have 
over : 

Number  of  personnel  hours  expended? 

Material  and  equipment  acquistion? 

Capital  investment? 

Energy  consumption? 

How  is  this  AIMD  evaluated  by  higher  authority? 

8.  What  primary  indicators  do  you  use  to  tell  you  that 
the  AIMD  is  doing  what  it  is  supposed  to  co? 

Do  you  have  a  standard  briefing  for  visitors?  If  sc. 
could  I  have  a  copv  of  your  briefing  and  si  ides°  Whac 
other  written  material  would  he  l  o  me  better  understand 
your  mission  ana  organizational  structure1 

10 .  What  else  shou.d  I  know? 
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APPENDIX  B:  IN-HOUSE  DOCUMENTATION 
Examples  of  Information  Reviewed  by  Facilitator 


1.  Manning  Documents/Manpower  Studies 

A.  Percent  authorized  manpower  onboard 

B.  Number  of  people  possessing  required  specialty 
training 

C.  Experience  level  of  personnel  onboard  (pay  grace) 

2.  Organizational  Charts 

A.  Department  Organization  Chart 

3 .  Division  Organization  Chart 
C.  Work  Center  Organization  Chart 

3.  Division  Briefing  Package 

4.  Monthly  Maintenance  Plans  for  Three  Months 

5.  Work  Center  Task  Description 

6.  Diagram  Showing  Flow  of  Components  Through  the  Repair 
Cycl e. 

7.  List  of  Number  of  Squadrons/Type  Aircraft  Supported 

8.  Instructions  and  Notices  that  govern  the  work  of  the 
Target  Organization 
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APPENDIX  C:  SITE  VISIT 

Examples  of  Items  Accomplished  During  a  Site  Visit 


NOTE 

If  the  visit  is  to  be  conducted  prior  to  the 
implementation  of  the  MGEEM .  then  the  facilitator  should  be 
introduced  as  a  visitor.  If  the  unit  personnel  have  been 
informed  about  the  purpose  and  events  associated  with  the 
MGEEM  implementation  process,  the  facilitator  should  be 
identified  as  being  associated  with  the  process. 


1.  Identify  the  target  organization's  major  products 
and  services. 

2.  Develop  a  general  understanding  of  the  work  flow. 

3.  Establish  a  rapport  with  the  workers  of  the  target 
organization  and  become  familiar  with  the 
organization's  structure. 

4.  Obtain  a  working  Knowledge  of  present  and  past 
history  of  the  work  center's  production  record/ 
problem  areas. 

5.  Become  familiar  with  the  units  that  target 
organization  has  to  depend  upon  to  do  its  job. 

6.  Become  familiar  with  the  customers  of  the  target 
organ i zat i on . 
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APPENDIX  D 
IMPLEMENTATION  PLAN 


APPENDIX,  D;  IMPLEMENTATION  PLAN 
Examples  of  Items  Included  on  an  Implementation  Plan 


NQXE 

After  the  Measurement  Plan  Is  developed.  It  should  be 
communicated  over  a  short  period  of  time  to  all  members  of 
the  target  organization  from  the  top  down.  The  communication 
should  De  In  the  form  of  both  face-to-face  and  written 
communication.  The  sequence  of  communication  activities  for 
a  branch  Is  outlined  below. 


08QQ-  (.Work. -Center  Super v  1  sors_me£t  lng ; .  The  division 
officer  Introduces  the  facilitator  and  discusses  the  purpose 
of  the  MGEEM  measurement  activity  within  the  work  center. 

The  facilitator  discusses  the  steps  Involved  In 
implementation  and  proposes  a  timetable.  The  work  center 
supervisor  Is  asked  for  comments,  and  schedule  conflicts  are 
resolved.  The  division  officer  and  work  center  supervisor 
agree  on  the  appropriate  time  for  the  facilitator  to  visit 
the  work  center  to  present  the  measurement  plan. 

0900  (Facilitator  and  Supervisor  finalize  Memorandum) . 

A  memorandum  Is  preparea  for  later  distribution  to  personnel, 
summarizing  the  highlights  of  the  implementation  schedule. 

The  memorandum  should  focus  on  the  purpose  of  the  measurement 
activity,  the  schedule,  and  how  members  of  the  organization 
will  be  affected. 

I-3Q.CL  tQ  1400  (Work  center  meeting).  The  facilitator  and 
division  officer  go  to  the  work  center  and  hold  a  brief 
meeting  with  work  center  members.  During  this  meeting, 
members  hear  the  division  officer,  division  chief,  and  work 
center  supervisor  express  support  for  the  MGEEM  measurement 
process.  The  facilitator  gives  a  general  outline  of  the 
steps  involved  in  implementation  and  what  will  be  expected 
from  each  member  of  the  organization.  Each  memoer  of  the 
work  center  receives  a  copy  of  the  previously  develoDea 
memorandum,  which  summarizes  the  information  presented. 
Members'  questions  are  answered. 
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INITIAL  BRIEFING  ON  KRA  DEVELOPMENT 


INITIAL  BRIEFING  ON  KRA  DEVELOPMENT 


APPENDIX  E= 

Introduction 

Introduction  of  Facilitator  and  Participants 
Purpose  of  the  Measurement  Activity  (Senior  Officer) 
Overview  of  MGEEM  Process 

Perspectives  on  Defining  Measures  (Slide  1) 

What  are  Key  Result  Areas?  (Slide  2) 

Purpose  of  This  Meeting 
Quest l ons 

Productivity  Overview 

Discussion  of  Organization  Diagram  (Figure  2.  Chapter  3 
made  l nto  a  slide) 

Definition  of  Productivity  for  the  Target  Organization 


Defining  Measures  -  Perspectives  (Slide  1  example) 

.  MEASUREMENT  IS  NOT  AN  END  IN  ITSELF 
.  WHAT  YOU  MEASURE  IS  WHAT  YOU  GET 
.  MEASUREMENT  SYSTEMS  FAIL  IF  THEY  LACK  ACCEPTANCE 
.  MEASUREMENT  IS  A  SUBSTITUTE  FOR  DIRECT  OBSERVATION 
.  MEASUREMENT  DATA  HAVE  THEIR  LIMITATIONS 
.  MOST  ORGANIZATIONAL  UNITS  CAN  BE  MEASURED 


Key  Result  Areas  are:  (Slide  2 _ Exampl.g). 

.  AREAS  REQUIRING  HIGH  LEVELS  OF  PERFORMANCE 

.  WAYS  TO  MEASURE  MISSION  EFFECTIVENESS 

.  THE  CRITICAL.  MAKE-OR-BREAK  AREAS  OF  A  JOB 

.  RESULTS:  NOT  ACTIVITIES.  PROCESS.  OR  PROCEDURES 

.  OUTPUTS.  NOT  INPUTS:  ENDS.  NOT  MEANS:  WHAT. 

NOT  HOW. 
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CONDENSED  LIST  OF  KRAS 


APPENDIX  F:  CONDENSED  LIST  OF  KRAs 


1.  Realistic  managerial  goals  In  accordance  with 
OPNAVINST  4790. 2D 

2.  Customer  satisfaction  through  quantity  and  quality 
gear  to  keep  aircraft  flying. 

3.  Motivated  work  force  with  high  morale. 

4.  Trained  personnel. 

5.  Optimum  utilization  of  production  personnel. 

6.  Safety 

7.  Maintain  real  property. 


169 


APPENDIX  G 


POSTING  OF  ME  CHARTS 


APPENDIX  G:  POSTING  OF  ME  CHARTS 
Overall  Mission  Effectiveness  Chart 


Composite  Mission  Effectiveness  Charts  for  KRAs  1-4 
KRA  #1  KRA  #2  KRA  #3  KRA  «4 


Individual  Indicator  charts  posted  by  KRA 
KRA  #1  KRA  #2  KRA  #3  KRA  #4 
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TWO  SAMPLES  OF  ME  CHARTS 


IUSSON  EJTtCTHtftSS  UIS30H 


APPENDIX  H 


TWO  SAMPLES  OF  ME  CHARTS 


PERCENT  OP  VORK  CENTER  PRODUCTION  EFFORT 
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MORALE  SURVEY 


APPENDIX  I  :  MORALE  SURVEY 


tf/C  6(0  Survey 


Instruct  ions : 


"or  each  st  Jt'ixnt  below,  mark  ao  "  X”  in  the  box 
corre s pood iag  co  cbe  answer  chat  most  closely  io* 
dicates  your  response  co  how  you  feeL  about  your 
work  environment. 


Statement 

1.  The  chance  co  do  something 
chat  makes  use  of  my  abilities. 

2.  The  way  the  Navy  poLicies  are 
puc  into  practice. 

3.  The  freedom  to  use  my  own 
judge menc . 

4.  The  chance  to  try  my  owo  methods 
of  doing  work. 

5.  The  working  conditions. 

6.  The  praise  C  get  for  doiog  a 
good  job. 

7.  The  feeling  of  accomplishment 
I  get  from  the  job. 


Very 


Very  Not  Dir-  Dis¬ 
satisfied  Satisfied  Sure  satisfied  satisfied 
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( continued  ) 


i . 


A  spirit  of  teamwork  exists 
betweeo  Che  people  in  ay  V/C. 


2.  A  spirit  of  teamwork  exists 
betweeo  the  people  io  ay 
division. 


3.  The  people  in  the  tf/C  work 

together  to  accomplish  the  V/C*s 
objectives. 


A.  the  people  ia  the  divisioo  work 
together  to  accomplish  the 
division's  objectives. 


5.  I  feel  a  lease  of  pride  at  being 
a  member  of  this  tf/C 


6.  t  feel  a  sense  of  pride  at  being 
a  member  of  this  division. 


I.  I  plan  to  reenlist  or  extend. 


2.  I  would  Like  to  leave  the  Mavy 
next  year. 


Strongly  Strongly 
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Strongly  Strongly 

Agree  Agnes  ttaimL  Disagree  Disagree 
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